.. _symbolication: ============= Symbolication ============= .. contents:: :local: Background ========== The Mozilla Symbolication Server (``_) has a symbolication API for converting memory addresses to symbols using debug information generated by builds of Mozilla products that were uploaded to the `Mozilla Symbols Server `__. .. Note:: 2021-10-07 The old symbolication API service hosted on symbols.mozilla.org is deprecated. ================== ========================================================= Status URL ================== ========================================================= Old and deprecated https://symbols.mozilla.org/symbolicate/v5 New https://symbolication.services.mozilla.com/symbolicate/v5 ================== ========================================================= Please update to the new service. Stacks ------ The stack is a list of lists. Each frame in the stack is composed of two things: 1. the index in the modules list for the module for this frame 2. the memory offset relative to the base memory address for that module as an base-10 integer For example: .. code-block:: text [1, 65802] is the base memory address 65,802 (0x1010a in hex) for the 1-index module in the ``memoryMap``. Modules ------- A module is something loaded into memory. It can be a library, binary, font, etc. We need to get the debug information for a given module in order to know what symbols correspond to which addresses. For Breakpad SYM files, modules are denoted by a debug filename and debug id. Each debug filename and debug id pair corresponds to a path in the symbols store. The full URL comes from taking the base url, appending the debug filename, appending the debug id, and then appending the debug filename again with a ``.sym`` extension. For example: .. code-block:: text ["wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2"] becomes: .. code-block:: text https://example.com/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym ^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^ | | debug filename with .sym extension | debug id debug filename What is Symbolication? ---------------------- Given a set of modules and a set of stacks, *symbolication* is the act of fetching the debug symbols files for the modules and then looking up the symbols corresponding to the module offsets. For eaxmple, given this set of modules: .. code-block:: ["firefox.pdb", "5F84ACF1D63667F44C4C44205044422E1"], ["mozavcodec.pdb", "9A8AF7836EE6141F4C4C44205044422E1"], ["Windows.Media.pdb", "01B7C51B62E95FD9C8CD73A45B4446C71"], ["xul.pdb", "09F9D7ECF31F60E34C4C44205044422E1"], ... and this stack: .. code-block:: [3, 6516407], [3, 12856365], [3, 12899916], [3, 13034426], ... you might end up with something like this: .. code-block:: text 0 xul.pdb mozilla::ConsoleReportCollector::FlushReportsToConsole(unsigned long long, nsIConsoleReportCollector::ReportAction) 1 xul.pdb mozilla::net::HttpBaseChannel::MaybeFlushConsoleReports()", 2 xul.pdb mozilla::net::HttpChannelChild::OnStopRequest(nsresult const&, mozilla::net::ResourceTimingStructArgs const&, mozilla::net::nsHttpHeaderArray const&, nsTArray const&) 3 xul.pdb std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1001:11',void>::_Do_call() ... Tips for Symbolication ====================== If you're trying to symbolication large numbers of things and it's going slowly, here are some things you can try: 1. **Always set the User-Agent.** Always set the user agent. If possible, include the url to the git repository where your code is stored. 2. **Wait and retry on HTTP 429 and HTTP 5xx responses.** If you get an HTTP 429 or HTTP 5xx response, wait and retry again. You could use a backoff of (0s, 1s, 2s, 3s, 4s, 5s). You should always get back a JSON response. If you don't, treat that like a temporary failure, wait a bit and try again. 3. If you're getting a 200 response, but some frames aren't symbolicated, then either there are no debugging symbols available for that module or the debugging symbols for that module are malformed. You can see if we have debugging symbols and check to see if they're well-formed by requesting the module from `Mozilla Symbols Server `__. 4. **Batch symbolication requests as a single request with multiple jobs.** If you batch your symbolication requests it'll reduce the HTTP request/response overhead and improve the likelihood that the request handler can take advantage of the symcache LRU cache. Further, it's best to batch requests for the same platform/buildid/product. That'll reduce the number of times the request handler has to parse the xul module sym file to at most 1. For example, if you were trying to symbolicate 1,000,000 crash pings from yesterday, you should: 1. sort them by platform/buildid/product 2. for each platform/buildid/product tuple: 1. for each N crash pings 1. create a symbolication request 2. perform the symbolication request Currently, a good `N` is 1,000, but it depends on the size of the stacks and what other modules are involved. You may find a lower `N` is better. If this isn't helpful, we want to know! Please `write up a bug and tell us `__. Symbolication: /symbolicate/v5 ============================== .. http:post:: /symbolicate/v5 :synopsis: Symbolicates stacks using Breakpad sym files. Symbolicate one or more jobs. Each job consists of one or more stacks and a memory map. Send an HTTP POST request with a JSON payload. Set a User-Agent. For example:: my-script/1.0 (+https://example.com/path-to-code/) For documentation on setting useful user agents: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings **Example request:** .. sourcecode:: http POST /symbolicate/v5 HTTP/1.1 User-Agent: my-script/1.0 (+https://example.com/path-to-code) { "jobs": [ { "stacks": [ [ [ 1, 7556668 ], [ 1, 7509754 ] ] ], "memoryMap": [ [ "firefox.pdb", "C0A5F7D110D262364C4C44205044422E1" ], [ "xul.pdb", "0FBE970321AB8CF14C4C44205044422E1" ] ] } ], "version": 5 } **Example response:** [#prettyresponse]_ .. sourcecode:: http HTTP/1.1 200 OK Content-Type: application/json { "results": [ { "stacks": [ [ { "frame": 0, "module": "xul.dll", "module_offset": "0x734e3c", "function": "mozilla::dom::JSActorManager::ReceiveRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe&&, mozilla::Maybe&&)", "function_offset": "0x7dc", "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/jsactor/JSActorManager.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e", "line": 172 }, { "frame": 1, "module": "xul.dll", "module_offset": "0x7296fa", "function": "mozilla::dom::WindowGlobalChild::RecvRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe const&, mozilla::Maybe const&)", "function_offset": "0x1ba", "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/WindowGlobalChild.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e", "line": 586 } ] ], "found_modules": { "firefox.pdb/C0A5F7D110D262364C4C44205044422E1": null, "xul.pdb/0FBE970321AB8CF14C4C44205044422E1": true } } ] } .. [#prettyresponse] The example response is indented for readability. Here's an example you can copy and paste--though symbols do age out of our system, so this may not be all that exciting: .. code-block:: shell curl --user-agent "my-script/1.0 (+https://example.com/)" \ -d '{"jobs": [{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]]}]}' \ https://symbolication.services.mozilla.com/symbolicate/v5 :json results: array of result objects--one for every job :[].stacks: array of symbolicated stacks where each stack is an array of JSON objects :frame (int): frame index; 0-based :module (str): the module name :module_offset (str): the module offset in hex :function (str): (optional) the function name :function_offset (str): (optional) the function offset in hex :file (str): (optional) the source file :line (int): (optional) the line number in the source file :inlines (array): (optional) an array of inline frames, each of which has optional "function", "file" and "line" properties, ordered from deepest call to most shallow :[].found_modules: json object indicating which modules we had symbols for and which ones we didn't :/ (str): `true` if we found symbols, `false` if we didn't, and `null` if we didn't need to look up symbols because it's not referenced in the stacks :reqheader User-Agent: please provide a unique user agent to make it easier for us to help you debug problems For example:: my-script/1.0 (+https://example.com/) See the MDN docs on good user agent practices: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings :reqheader Debug: if you add ``Debug: true`` to the headers, then symbolication will also return debug information about cache lookups, how many downloads, timings, and some other things :statuscode 200: success symbolicating stacks :statuscode 429: your request was rate-limited; wait a bit and retry [#ratelimiting]_ :statuscode 500: something bad happened--please open up a bug :statuscode 503: temporary problem with the server; wait a bit and retry .. [#ratelimiting] Eliot isn't currently rate-limiting, but it's something we could add in the future, so it's best for you to handle it now. Symbolication: /symbolicate/v4 (deprecated) =========================================== .. http:post:: /symbolicate/v4 :deprecated: :synopsis: Symbolicates stacks. Symbolicate one or more stacks. Send an HTTP POST request with a JSON payload. .. Warning:: Don't use Symbolication v4 for anything new. Please migrate to v5 as soon as you can. v4 will be removed in the near future. Follow this bug for status: https://bugzilla.mozilla.org/show_bug.cgi?id=1475334 **Example request:** .. sourcecode:: http POST /symbolicate/v4 HTTP/1.1 { "memoryMap": [ [ "xul.pdb", "44E4EC8C2F41492B9369D6B9A059577C2" ], [ "wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2" ] ], "stacks": [ [ [0, 11723767], [1, 65802] ] ], "version": 4 } **Example response:** .. sourcecode:: http HTTP/1.1 200 OK Content-Type: application/json { "symbolicatedStacks": [ [ "XREMain::XRE_mainRun() (in xul.pdb)", "KiUserCallbackDispatcher (in wntdll.pdb)" ] ], "knownModules": [ true, true ] }