Implement initial "waiting" logs with tqdm? #327

Activity

1. tswast commented on Sep 11, 2020

   

   tswast

   on Sep 11, 2020

   Contributor

   More actions

   We don't know ahead of time how long a query will take, so the current tqdm logic won't work.

   Looks like there are a couple of open issues for indefinite progress bars at: tqdm/tqdm#427 tqdm/tqdm#925

   A couple of options:

   • Add "spinner" feature to tqdm and use that.
   • Do some kind of exponential backoff on "waiting..." logging. 1/s at first, ramping up to 1/min?
   • Use print statements with the right console codes to rewrite lines in-place to update the elapsed time instead of logging.
2. max-sixty commented on Sep 12, 2020

   

   max-sixty

   on Sep 12, 2020

   ContributorAuthor

   More actions

   Good point @tswast

   I just looked back on some code I wrote a few years ago and found this — coincidentally for waiting for jobs from Google Cloud! — though now I see the links above, maybe I should be more cautious about suggesting it's easy to have an indefinite progress bar...

   def wait_for_job(job, timeout_in_seconds=None):
       # https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigquery/cloud-client/snippets.py
       if timeout_in_seconds:
           start = datetime.datetime.now()
           timeout = start + datetime.timedelta(0, timeout_in_seconds)
   
       with tqdm(
           bar_format="Waiting for {desc} Elapsed: {elapsed}", total=10000
       ) as progress:
           while True:
               job.reload()  # Refreshes the state via a GET request.
               progress.set_description(str(job))
               if job.state == "DONE":
                   if job.error_result:
                       raise RuntimeError(job.errors)
                   progress.bar_format = "Completed {desc}. Elapsed: {elapsed}"
                   return
               if timeout_in_seconds:
                   if datetime.datetime.now() > timeout:
                       raise SystemError(f"Timed out after {timeout_in_seconds} seconds")
               time.sleep(1)

   😄React with 😄1Reacted by Tim Sweña (Swast)
3. tswast

   added

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   on Oct 6, 2020
4. tswast commented on Oct 26, 2020

   

   tswast

   on Oct 26, 2020

   Contributor

   More actions

   I had some more thoughts about this. The UI uses the job statistics to show progression through the various stages. Filed googleapis/python-bigquery#343 to see if we can implement this in google-cloud-bigquery, since it'll be relevant for the %%bigquery magics, too.

   🎉React with 🎉1Reacted by Maximilian Roos
5. tswast commented on Nov 16, 2020

   

   tswast

   on Nov 16, 2020

   Contributor

   More actions

   I just merged googleapis/python-bigquery#352 which will be available in google-cloud-bigquery 2.4.0 googleapis/python-bigquery#381
6. product-auto-label

   added

   api: bigqueryIssues related to the googleapis/python-bigquery-pandas API.Issues related to the googleapis/python-bigquery-pandas API.

   on Jul 17, 2021
7. meredithslota

   closed this as completedon Aug 23, 2023