Skip to content

[PyTorch] move from input ivalues in ByteCodeDeserializer #64029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 6 commits into from
Closed

[PyTorch] move from input ivalues in ByteCodeDeserializer #64029

wants to merge 6 commits into from
+86 −62

Conversation

swolchok
Copy link
Contributor

@swolchok swolchok commented Aug 26, 2021 •
edited
Loading

@swolchok
[PyTorch] move from input ivalues in ByteCodeDeserializer
ca99b85 
This should save us a separate pass over the data structure to destroy it.

Differential Revision: [D30559530](https://our.internmc.facebook.com/intern/diff/D30559530/)

[ghstack-poisoned]
@facebook-github-bot facebook-github-bot added oncall: jit Add this issue/PR to JIT oncall triage queue cla signed labels Aug 26, 2021
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Aug 26, 2021 •
edited
Loading

🔗 Helpful links

💊 CI failures summary and remediations

As of commit ca7a71e (more details on the Dr. CI page):

  • 1/2 failures introduced in this PR
  • 1/2 broken upstream at merge base 6d5f33d on Aug 26 from 9:52am to 10:07am

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See GitHub Actions build linux-bionic-py3.8-gcc9-coverage / test (distributed, 1, 1, linux.2xlarge) (1/1)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-09-08T16:05:00.6019328Z test_udf_remote_...yUniqueId(created_on=0, local_id=0) to be created. 
2021-09-08T16:04:19.9787947Z frame #15: <unknown function> + 0x486ea (0x7f83227326ea in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
2021-09-08T16:04:19.9789259Z frame #16: <unknown function> + 0xc9039 (0x7f832263e039 in /opt/conda/lib/libstdc++.so.6)
2021-09-08T16:04:19.9790651Z frame #17: <unknown function> + 0x76db (0x7f83461606db in /lib/x86_64-linux-gnu/libpthread.so.0)
2021-09-08T16:04:19.9791953Z frame #18: clone + 0x3f (0x7f8345e8971f in /lib/x86_64-linux-gnu/libc.so.6)
2021-09-08T16:04:19.9792525Z 
2021-09-08T16:04:20.3749719Z ok (3.724s)
2021-09-08T16:04:35.6261654Z   test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... ok (15.248s)
2021-09-08T16:04:44.8611174Z   test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... ok (9.238s)
2021-09-08T16:04:48.5858901Z   test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... ok (3.725s)
2021-09-08T16:04:56.3180655Z   test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... ok (7.732s)
2021-09-08T16:05:00.6019328Z   test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... [E request_callback_no_python.cpp:559] Received error while processing request type 261: falseINTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":385, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2021-09-08T16:05:00.6021857Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:385 (most recent call first):
2021-09-08T16:05:00.6024494Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x59 (0x7f0c30e16f59 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
2021-09-08T16:05:00.6026280Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xa3 (0x7f0c30dedb34 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
2021-09-08T16:05:00.6028295Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x61 (0x7f0c30e14341 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
2021-09-08T16:05:00.6030199Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x628 (0x7f0c3a34bd38 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
2021-09-08T16:05:00.6032746Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x8c (0x7f0c3a33258c in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
2021-09-08T16:05:00.6037455Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xf5 (0x7f0c4abed9a5 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
2021-09-08T16:05:00.6040426Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x1f0 (0x7f0c3a339120 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
2021-09-08T16:05:00.6042873Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x60 (0x7f0c4abed270 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
2021-09-08T16:05:00.6044529Z frame #8: <unknown function> + 0x92aa0d0 (0x7f0c3a32e0d0 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD
This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

This was referenced Aug 26, 2021
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@swolchok
Update on "[PyTorch] move from input ivalues in ByteCodeDeserializer"
91cbf76 
This should save us a separate pass over the data structure to destroy it.

Differential Revision: [D30559530](https://our.internmc.facebook.com/intern/diff/D30559530/)

[ghstack-poisoned]
@swolchok swolchok mentioned this pull request Aug 30, 2021
Closed
@swolchok
Update on "[PyTorch] move from input ivalues in ByteCodeDeserializer"
293ccca 
This should save us a separate pass over the data structure to destroy it.

Differential Revision: [D30559530](https://our.internmc.facebook.com/intern/diff/D30559530/)

[ghstack-poisoned]
This was referenced Aug 31, 2021
Closed
Closed
@swolchok
Update on "[PyTorch] move from input ivalues in ByteCodeDeserializer"
5eb1c87 
This should save us a separate pass over the data structure to destroy it.

Differential Revision: [D30559530](https://our.internmc.facebook.com/intern/diff/D30559530/)

[ghstack-poisoned]
@swolchok
Update on "[PyTorch] move from input ivalues in ByteCodeDeserializer"
4a5690f 
This should save us a separate pass over the data structure to destroy it.

Differential Revision: [D30559530](https://our.internmc.facebook.com/intern/diff/D30559530/)

[ghstack-poisoned]
@swolchok
Update on "[PyTorch] move from input ivalues in ByteCodeDeserializer"
ca7a71e 
This should save us a separate pass over the data structure to destroy it.

Differential Revision: [D30559530](https://our.internmc.facebook.com/intern/diff/D30559530/)

[ghstack-poisoned]
@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 0d0d2f2.

@facebook-github-bot facebook-github-bot deleted the gh/swolchok/275/head branch September 12, 2021 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cla signed Merged oncall: jit Add this issue/PR to JIT oncall triage queue
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@swolchok @facebook-github-bot