Log profiling information from worker. by robertnishihara · Pull Request #178 · ray-project/ray

robertnishihara · 2017-01-05T05:44:16Z

Note: this slows down the simple benchmark below by about 40us.

@ray.remote
def f(x):
  return x

%timeit ray.get(f.remote(3))

I think about 5 or so microseconds come just from the additional IPC to photon to log the events (though that call isn't on the critical path, so it's not clear why it slows it down with multiple workers (but it does). The remaining slowdown comes from all of the logging calls.

For example, a simple logging call like the following takes around 4us, and we do that around 8 times per task, which seems to explain the remaining slowdown.

def g():
  with ray.log_span("event_type1"):
    pass

%timeit g()

We'll need to redo the logging code to address this.

pcmoritz · 2017-01-05T06:01:35Z

+
+  Args:
+    event_type (str): The type of the event.
+    contents: More general data to store with the event.


let's demand here that content is a little more restricted, say dicts with only strings as keys and vals, to make it trivially json serializable

pcmoritz · 2017-01-05T06:01:52Z

+
+def flush_log(worker=global_worker):
+  """Send the logged worker events to the global state store."""
+  worker_id_hex = repr(worker.worker_id)[9:-1]


needs fixing

pcmoritz · 2017-01-05T06:04:04Z

+    offset += payload_length;
+    CHECK(offset == length);
+    if (state->db != NULL) {
+      ray_log_event(state->db, key_name, key_name_length, payload,


maybe print a warning if not connected here? (together with the message so it will not get lost)

pcmoritz · 2017-01-05T06:08:19Z

+                      int64_t key_name_length,
+                      uint8_t *payload,
+                      int64_t payload_length) {
+  int64_t message_length = sizeof(key_name_length) + sizeof(payload_length) +


let's put a TODO here that this should be flatbufferized (then we can also pass the flatbuffers through from python directly)

pcmoritz · 2017-01-05T06:08:53Z

  case TASK_DONE: {
  } break;
+  case EVENT_LOG_MESSAGE: {
+    /* Parse the message. */


let's put a flatbuffers todo here too

## Description Bumps `sphinxcontrib-openapi`, which renders the [Ray Jobs REST API specification](https://docs.ray.io/en/master/cluster/running-applications/job-submission/api.html) page, from 0.9.0 to [0.10.0](https://pypi.org/project/sphinxcontrib-openapi/0.10.0/), and regenerates the docs deplock. 0.10.0 fixes five upstream bugs, two of which this page's source comment named as blockers for request and response examples: - [sphinx-contrib/openapi#166](sphinx-contrib/openapi#166): `:generate-examples-from-schemas:` crashed on an `allOf` that targets a non-object schema, which is how `JobDetails` attaches descriptions to the `JobType` and `JobStatus` enums. - [sphinx-contrib/openapi#165](sphinx-contrib/openapi#165): `:response-examples-for:` and the other whitespace-delimited options were rejected as unknown options. - [#173](sphinx-contrib/openapi#173), [#171](sphinx-contrib/openapi#171), and [#178](sphinx-contrib/openapi#178): an operation summary written as a YAML block scalar rendered literal `**`, the old renderer's request body block was left unclosed, and the 3.0 renderer crashed on a non-object request body. Examples still can't be enabled. With `:generate-examples-from-schemas:` on 0.10.0, the stop endpoint's 200 response, a bare `JobStatus` enum, renders as `PENDING` rather than `"PENDING"` under `Content-Type: application/json`. The `http` lexer rejects it, and the warning fails the build. That's [sphinx-contrib/openapi#172](sphinx-contrib/openapi#172), fixed for the `httpdomain` renderer on the open [sphinx-contrib/openapi#185](sphinx-contrib/openapi#185). This PR updates the source comment in `api.md` to name that bug instead of the two that are now fixed. The rendered page doesn't change. What changes: - `doc/requirements-doc.txt`: `sphinxcontrib-openapi==0.9.0` → `0.10.0`. - `python/deplocks/docs/docbuild_depset_py3.11.lock`: regenerated with raydepsets. The diff is the one package block. 0.10.0 declares the same dependencies as 0.9.0, so there's no transitive churn. - `doc/source/cluster/running-applications/job-submission/api.md`: source comment only. 0.10.0 raises `requires-python` from `>=3.10` to `>=3.11`. The only consumer is the docbuild depset, which is Python 3.11 on both Read the Docs and the `docbuild-py3.11` CI image. ## Related issues Related to sphinx-contrib/openapi#172. ## Additional information **Not a duplicate.** No open PR touches `sphinxcontrib-openapi` or bumps this pin, and no open Dependabot PR targets `doc/requirements-doc.txt`. **Tests run:** - `bazelisk run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/docs.depsets.yaml --name docbuild_depset_311` regenerated the lock. - `bazelisk run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/docs.depsets.yaml --check` passed for all three docs depsets. The doctest lock is unchanged. - `make -C doc rtd-build` in a Python 3.11 venv installed from `doc/requirements-doc.lock.txt`: `build succeeded.`, zero Sphinx warnings under `-W`. The `rtd-doctor` preflight confirmed the environment matches Read the Docs. The rendered Jobs API page has all 8 endpoints, including `/stop`. - The Jobs spec rendered standalone with `openapi_default_renderer = "httpdomain"` under `-W`: clean on 0.10.0 with no example options. With `:generate-examples-from-schemas:` it fails on the unquoted `PENDING`, which is the bug the updated comment describes. AI assistance (Claude Code) was used to prepare this change. I reviewed every changed line. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Douglas Strodtman <douglas@anyscale.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

robertnishihara added 2 commits January 4, 2017 18:30

Log timing events on workers.

b446658

Have workers log to the event log through the local scheduler.

7add17e

pcmoritz reviewed Jan 5, 2017

View reviewed changes

robertnishihara and others added 3 commits January 5, 2017 01:21

Fixes and address comments.

5b00168

bug fix

db37bfa

styling

91d8700

pcmoritz merged commit 651aa60 into ray-project:master Jan 6, 2017

pcmoritz deleted the eventlog branch January 6, 2017 00:47

Navigation Menu

Repository navigation

Log profiling information from worker. - #178

Log profiling information from worker.#178
pcmoritz merged 5 commits into
ray-project:masterfrom
robertnishihara:eventlog

robertnishihara commented Jan 5, 2017 •

edited

Loading

Uh oh!

pcmoritz Jan 5, 2017

Uh oh!

pcmoritz Jan 5, 2017

Uh oh!

pcmoritz Jan 5, 2017 •

edited

Loading

Uh oh!

pcmoritz Jan 5, 2017

Uh oh!

pcmoritz Jan 5, 2017

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

robertnishihara commented Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pcmoritz Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

pcmoritz Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

pcmoritz Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcmoritz Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

pcmoritz Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robertnishihara commented Jan 5, 2017 •

edited

Loading

pcmoritz Jan 5, 2017 •

edited

Loading