Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info

  1. Request submitted by the user and arrives in the the Web-API queue

    • The request arrives to the service back-end and a new record is added in the service database. (MongoDB).

    • At this phase the request gets the status "queued" and it becomes visible in API jobs list but it is not yet visible on the MARS activity

    • (lightbulb) Please note:

      • Each request corresponds to one record in MongoDB. 

      • All the request's details (status, timers, host, user etc)  are stored in MongoDB.

      • The request record is updated periodically to reflect  its current status.

    • The "queueing time" depends on several factors, ie priorities, limitations, current load etc

    • (lightbulb)Please note that  to improve the service some QoS limitations and priorities are applied, like:

      • The total maximum number active requests

      • The maximum number of active requests per user

      • A MAX Nr of  Web API active slots is allowed.

      • etc
    • Once an API slot is available and if all the QoS rules are met, the request will become "active"  on the API level.

  2. Request becomes active on API level

    1. Split: On the API level,  the request is split into a number of smaller requests to achieve a better performance from the MARS point of view.

      1. For instance if a request is requesting one year of "ERA interim" daily data it is split per month. ie it is split into 12 requests.

    2. Validate: Each of these sub-requests are validated and passed to a MARS client running on one of the "atlas" hosts. The atlas machines are visible in MARS activity

    3. Create file: Once all the sub requests have completed successfully the corresponding sub-files are merged into a final one and the request status becomes complete

      1. If something goes wrong during the procedure above the request is "aborted"

  3. Request becomes active on MARS level

    • At this phase the request  becomes visible on the MARS activity (with status active or queued in MARS)

    • Note that a request can be at status active on the level of API but queued on MARS.

    • We always try to tune the Web-API and MARS systems in order to improve the performance but it most cases it is not so straightforward.

    • On the level of MARS server there are several MARS QoS rules (ie limitations, restrictions and priorities) like the ones below.

      • On marsod-core, the total number of Requests from ECMWF Web API with 1 tape mount [source] is limited to 6

      • On marsth-core, the total number of Requests from ECMWF Web API with 1 tape mount [source] is limited to 12

      • To avoid long queuing times it would be better is strongly recommended to improve the efficiency of your requests and try to split your long period requests into smaller ones. Read Retrieval efficiency

  4. File transfer procedure

    • The produced file is then transferred by the Web-API client from ECMWF proxy to the user's local disk.

    • Once the file has been transferred successfully the file will be removed from our disks and it won't be any more available.

If transfer procedure fails
Info
  1. The file transfer procedure may fail due to several reasons (eg network connectivity problems between our end and the users) see more here:Troubleshooting#asserttotal==sizeAssertionError)

  2. If  transfer procedure fails difficult to trace back what has happened (warning)
  3. The API client checks the volume of the transferred file and if the size is not the expected one the transfer procedure will start again. The MAX Nr of retries is 10 I

...