Assertion failure editing invalid UTF-32 #11877
Labels
bug
issues reporting wrong behavior
complexity:low
Low-risk, unlikely to cause distant/indirect damage
encoding
good-first-issue
community: issues for first-time contributors
status:has-plan
Milestone
Comments
justinmk
added
complexity:low
|
I imagine the correct approach here would actually be to use The worst that could happen is that on platforms that actually do throw a signal on cast overflow/underflow (do we support them in the first place?), SIGABRT or similar will be thrown.. which the current code does for all platforms when the assert fails, so it'll be strictly an improvement. |
What does this even mean?
Before getting stuck in the same obsession about casting, assertions, checked conversions etc shouldn't we consider the actual semantics of the code and the end goals that it is trying to accomplish? What the code does is converting a codepoint to UTF-8. The limit of a codepoint is defined to be 0x10FFFF (and not what the limit of some int type happens to be), but because $REASONS vim still handles values up to 0x7FFFFFFF. Beyond that vim seems to silently wrap around values (to random unrelated codepoints). The proper handling of invalid codepoints (if we don't want to error out the entire file read, which I think we don't) is to change them to the replacement char U+FFFD �. Ideally we would replace all values over 0x10FFFF ( #6325 ), but as to fix the conversion issue, we could replace values above INT_MAX to 0xFFFD already. Perhaps there is some flag that also should be set that the conversion was destructive (which it already is). |
|
Yes, but I'm not talking about the actual conversion here but rather the ingestion of unknown data that may or may not be valid. I'm not talking about the actual Unicode-aware logic but merely the mechanics of converting individual bytes to a Unicode codepoint. Given that you are going from n bytes to a may-or-may-not-be-valid-Unicode codepoint and you want to evaluate the validity (or lack thereof) of the maybe-codepoint, you can't assume that the up-to-four-byte sequence you read will lie in the 0x000000-0x10FFFF range. If you want to make your decision about whether or not a particular sequence of four bytes is valid Unicode after having created it (vs determining whether ingesting a particular, single byte will result in the codepoint being formed being out-of-range, which is just complicating things for no good reason) then you cannot assume that this sequence of bytes will fit in a signed 4-byte integer upon which you predicate your validity test. That said, I agree with everything else you said about lossy conversion and not aborting the process merely because an invalid codepoint was found, but I think we disagree about what the ideal behavior when an invalid codepoint is encountered should be. IMHO, in an ideal world and in a greenfield implementation if I load a binary file as UTF-8, I would like to still be able to save it in neovim without corrupting its original contents: e.g. there should be a difference between the lossy representation of a sequence of bytes in the editor and the actual underlying value. I believe the editor should show U+FFFD but preserve the actual values unless the user selects the � character and replaces it with something else. In fact, I think that I would prefer emitting invalid UTF-8 and having the terminal deal with it over actively destructive manipulation of the input bytestream (even if it causes neovim and the terminal to differ on the length (in on-screen cells) of invalid sequences which could cause many other problems.) |
Again, a codepoint doesn't fit into uint8_t. Do you mean uint32_t should be used? I don't think this helps very much because even vim:s improper-utf-8 is still 31 bits and not 32 bits.
This is about reading UTF-32. Not about reading UTF-8. We are reading exactly four bytes at a time here with no assumptions.
100% agree and #6325 preserves this, but this is a separate question from UTF-32 to UTF-8 conversion which was the code path discussed in this issue. Invalid UTF-32 cannot be fully preserved as only 31 bits can be represented and not 32. (you can of course edit it as binary, but then you get no conversion for valid chars). The code path in vim is already destructive, I just want to make it deterministically destructive (always replacement char, instead of the whims of the gods of Implementation Defined Behavior) and somewhat closer to "sane" unicode semantics even if not fully there there. |
|
Sorry, I had actually edited my comment but you were too quick for me. I was interrupted non-stop in writing it and ended up with a mess. |
|
I don't think it has to be destructive at all. Ideally edits can be converted into whatever codepage the file is specified (via fenc) as having (after all, replacing existing valid characters or adding new ones is no different than selecting the � marker and replacing that with valid characters) but unmodified content would be preserved. |
|
No worries, but let's focus on UTF-32 reading only for now, we can consider overlong UTF-8 in the linked PR and forget about it for now. The concrete thing I'm suggesting for this particular conversion issue is to replace values over INT_MAX with the replacement char. Any objection to that? |
Would an integration test that opens a binary file, then only calls |
|
it has to be destructive because vim/nvim cannot represent 2^32 codepoints, only 2^31. This is a fundamental limit that ain't gonna change anytime soon. I'm just saying we should use a deterministic � value as suggested by the unicode standard rather than what the C implementation happens to be giving us (as vim relies upon).
One destructive behavior is changed to another destructive behavior. I suppose it is subjective if this counts as a "regression" (spacebar heating). |
|
I thought you were saying "it is already destructive because vim uses signed types and that results in implementation-specific behavior" and was prepared to argue about theoretical vs actual destructive behavior but no, it's a lot more broken than that and it is/was completely destructive right now, regardless of platform or implementation-specific behavior. Swapping one destructive behavior for (a much more reasonable) another is perfectly fine with me. As far as I'm now concerned, the result of such an operation can be treated as (neo)vim's equivalent of undefined behavior and it's fair game to change it. |
|
Hi, I'd like to try working on fixing this issue, if possible. I replicated the bug. Could you confirm if my understanding of the problem is correct? What is going wrongUTF-8 is designed to handle up to 0x10FFFF, but vim for some reason handles values up to 0x7FFFFFFF Suggested solutionReplace values above INT_MAX to 0xFFFD |
|
The suggested solution is the correct one here, but "What is going wrong" is a separate issue, addressed by #6325 potentially. |
muniter
added a commit
to muniter/neovim
that referenced
this issue
Nov 18, 2021
muniter
added a commit
to muniter/neovim
that referenced
this issue
Nov 18, 2021
muniter
added a commit
to muniter/neovim
that referenced
this issue
Nov 24, 2021
gpanders
pushed a commit
that referenced
this issue
Nov 24, 2021
|
Thanks, @muniter! |
bfredl
added a commit
that referenced
this issue
Nov 30, 2021
BREAKING CHANGES * 32-bit windows builds are no longer provided. * **build deps**: use libuv 1.42.0 upstream for WIN32 (#15889) (f6c0a37), closes #15889 * removes Windows 7 support * removes support for mouse and alternate buffers in TUI for Windows 8 and 8.1 * **lsp/diagnostic:** highlight groups and signs for LSP diagnostics renamed (e.g. `LspDiagnosticsDefaultWarning` to `DiagnosticWarn`) (a5bbb93) * **diagnostic:** make DiagnosticChanged a first class autocmd (#16098) (150a592), closes #16098 * `au User LspDiagnosticsChanged` autocommands are not supported. Use the new first-class DiagnosticChanged event instead. * **lua**: `register_keystroke_callback` => `on_key` (#15460) (69fe427) * **note**: this breaking change was included in 0.5.1 FEATURES * **:source, nvim_exec:** defer script item creation until s:var access (da9b0ab) * **:source, nvim_exec:** support script-local variables (d4ed51e), closes #13143 #11507 * **lua:** add trimempty optional parameter to vim.split (5fa26e2) * **lua:** add vim.str_utf_{start,end} (#16129) (2230b57) * **lua:** add vim.str_utf_pos function (d752cbc) * **lsp:** aggregate code actions from all clients (#15121) (c36df20) * **api:** add lua C bindings for xdiff (#14536) (3d3c0c6) * **api:** evaluate statusline string #16020 (9086938) * **api:** named marks set, get, delete #15346 (49fdc62) * **api:** nvim_get_chan_info: include "argv" for jobs #15537 (0603eba), closes #15440 * **api:** win_viewport also sends line_count #15613 (086631c) * **api:** support :terminal input callback in lua (9e41e82) * **ci:** add backport PR action (#14766) (6cc456d) * **decorations:** allow more than one stacked highlight in a virt_text (1495d36) * **decorations:** support virtual lines (392c658) (8d7816c) * **diagnostic:** move vim.lsp.diagnostic to vim.diagostic and support other sources (a5bbb93) * **diagnostic:** add 'prefix' option to open_float (#16321) (3c74ba4) * **diagnostic:** add option to include diagnostic source (d43151e) * **diagnostic:** allow 'prefix' option to return highlight (cc48837) * **diagnostic:** allow customized diagnostic messages (#15742) (d999c96) * **diagnostic:** match(), tolist(), fromlist() #15704 (e61ea77) * **diagnostic:** update jumplist on goto_next/prev (#15942) (b55944e) * **diagnostic:** use `scope = 'line'` by default for `open_float()` (#16456) (217f9f8), closes #16453 * **diagnostic:** support severity_sort (32c0631) * **checkhealth:** provide function for command line completion (8b43b07) * **f_chansend:** support Blob data argument (7e9ea08) * **job:** add parameter to close stdin (eb7f24b) * **keywordprg:** use :terminal for external commands #15398 (a90513c), closes #2995 #2761 * **lsp:** add 'focus' option to open_floating_preview (#16465) (fff8827) * **lsp:** add a registry for client side code action commands (6c03601) * **lsp:** add client command support to codelens (#15820) (19a77cd) * **lsp:** add codeAction/resolve support (#15818) (ec4731d) * **lsp:** add exit_timeout flag (#16070) (80456cf) * **lsp:** add formatexpr (#16186) (52fa1d2) * **lsp:** add lsp healthcheck (e268026) * **lsp:** add per-client commands (#16101) (519d8de) * **lsp:** add tagfunc (#16103) (f940e7a) * **lsp:** add warning message for large log size (e6777a7) * **lsp:** allow configuring zindex for floating windows (#15086) (c487a73) * **lsp:** allow diagnostics to be disabled for a buffer (#15134) (1aeb945) * **lsp:** allow root_dir to be nil (#15430) (ff0833c) * **lsp:** highlight active parameter in signature help (#15018) (af26371) * **lsp:** improve vim.lsp.util.apply_text_edits (#15561) (41cfba6) * **lsp:** include original request params in handler ctx (187579f) * **lsp:** jump to diagnostics by position (#14795) (ea39ff5) * **lsp:** Make line diagnostics display prettier (e43dbfd) * **lsp:** make list handlers configurable (#15199) (3e00d4f) * **lsp:** support textDocument/prepareRename (#15514) (c1f573f) * **lsp:** use uv_spawn to check if server executable (#16430) (1a60580) * **lsp:** use vim.ui.select() in codelenses (#16004) (e7ea54a) * **lsp:** utilize textEdit.range for startbyte in omnifunc (#15957) (e9d6f7c) * **lua:** add lua-cjson as vendored dependency (8decc9f) * **lua:** add vim.mpack for msgpack support in lua (eaf661d) * **lua:** allow passing handles to vim.b/w/t (6c5e7bd) * **lua:** convert binary string with NULs to Blob (de9df82) * **lua:** document support of packages with v:lua syntax (9dd371b) * **lua:** enable stack traces in error output (#16228) (03b805a) * **lua:** expose lua-cjson as vim.json (30fed27) * **lua:** make vim.mpack support vim.NIL and vim.empty_dict() (0f59666) * **match:** allow hl group to be defined after :match command (fca52f5) * **msgpack:** convert Blobs to BIN strings (af6f454) * **health:** support lua healthchecks (9249dcd), closes #15632 * **shada:** restore Blob globals properly (ef729fb) * **terminal:** TermClose: set exit code in v:event.status #15406 (50b30de), closes #4713 * **treesitter:** add next, prev sibling method (1400841) * **treesitter:** allow to set highlight priority for queries (242608e) * **ui:** add `opt.kind` to `vim.ui.select` (#15838) (7ae86c1) * **ui:** add vim.ui.input and use in lsp rename (#15959) (16d4af6) * **ui:** add vim.ui.select and use in code actions (#15771) (63fde08) * **vim script:** support calling v:lua as a method (b2994e3) CHANGES * **defaults:** auto-create backup dir (4600193) * **defaults:** inccommand=nosplit #15395 (7215d35) * **defaults:** set undo points in <C-U> and <C-W> (#15400) (2cb8db3) * **defaults:** limit syntax cost on CmdwinEnter #15401 (622a36b), closes #6289 #6399 * **defaults:** map CTRL-L to search highlights, update diffs #15385 (0aa8128) * **defaults:** map Y to y$ #13268 (5a111c1), closes #416 #6289 * **defaults:** remove 'options' from viewoptions #15397 (3954537), closes #6289 * **defaults:** set hidden (f6c72b7) * **defaults:** set nojoinspaces (d417e67) * **defaults:** switchbuf=uselast #15394 (4ba7495) * **runtime:** add packages as `"/pack/*/start/*"` patterns to &rtp (9df7e02) * **startup:** load builtin plugins with --clean #15893 (c7a63f3), closes #15605 * **terminal:** set cursorlineopt=number in terminal mode (#15493) (c61a386) * **window:** skip non-focusable floats for :windo (#15378) (e8631cb) PERFORMANCE IMPROVEMENTS * **api:** avoid spurious allocations when converting small objects (705e8f1) * **highlight:** use a hashtable for highlight group names (bb4b4d7) * **lua:** optimize vim.deep_equal #15236 (4b452d4) * **lua:** don't use regexes inside lua require'mod' (ea2023f), closes #15147 #15497 * **lsp:** improve json deserialization performance (#15854) (912a6e5) * **map:** reduce double pointer indirection to single pointer indirection (9e651a9) * **treesitter:** avoid string lookup of highlight name in hot loop (2460f0a) FIXES * **:source, nvim_exec:** handle Vimscript line continuations #14809 (6188926), closes #14807 * **:source:** copy curbuf lines to memory before sourcing #15111 (afdc9e6) * allow str_utfindex second argument to be an explicit nil (#16448) (512ec46) * **api:** fix crash after set_option_value_for() #15390 (8b0e6cc), closes #14097 #13577 * **api:** fix nvim_buf_set_extmark (2338345) * **autocmd:** fix conditions in block_autocmds, unblock_autocmds #15372 (29712ae), closes #6279 * **buffer_updates:** cleanup test behavior (54b2c68) * **buffer_updates:** handle :delete of the very last line in buffer (8335e26) * **buffer_updates:** handle :sort of already sorted buffer (ef687d3) * **buffer_updates:** make `lockmarks` not affect extmarks and buffer updates. fixes #12861 (7d171b1) * **bufupdates:** send correct updates for visual paste (1423146) * **build:** add an env var to re-enable the colors (5087347) * **build:** call find_package(Threads) before using its variables (f446ab3) * **build:** export symbols on Windows (aa644b7) * **build:** fix build failure in MinGW (0503e17) * **build:** make vendored libmpack and libmpack-lua build properly (2a08aef) * **channel:** throw error if sending to internal channel w/o terminal (3b89fee) * **checkhealth:** duplicate checks if module name has "-" #15935 (a36c6e5) * **checkhealth:** mitigate issues with duplicate healthchecks #15919 (acd5e83), closes #15259 * **ci:** disable broken test on openbsd on all CI due to resource constraints (a3e2636) * **ci:** re-run GHA for ready_for_review events (#15377) (c6ef956) * **decorations:** crash when :bdelete (extmark_free_all) after clear_namespace (cd353aa), closes #15212 * **defaults:** "syntax sync maxlines=1" on CmdwinEnter #15552 (5f8518b), closes #15401 * **defaults:** do not map Y in visual-mode #15387 (54726e8), closes #13268 * **diagnostic:** allow floats to be focusable (#16093) (427bac6) * **diagnostic:** change default severity_sort order (938ed45) * **diagnostic:** clamp line numbers in setqflist and setloclist (5b0d8f8) * **diagnostic:** correctly handle folder level diagnostics (f87779a) * **diagnostic:** deepcopy diagnostics before clamping line numbers (2abc799) * **diagnostic:** do not focus floats in goto functions (#16433) (b5b025f) * **diagnostic:** don't clamp line numbers in setqflist (0341c68) * **diagnostic:** don't return nil when callers expect a table (#15765) (057606e) * **diagnostic:** don't use nil col if missing from qflist (#16357) (5e46f64) * **diagnostic:** error on invalid severity value (#15965) (d5dd0aa) * **diagnostic:** fix navigation with diagnostics placed past end of line (34bb5fa) * **diagnostic:** fix option resolution in open_float (#16229) (fd34784) * **diagnostic:** fix wrong data type in setqflist() (3fd1450) * **diagnostic:** get line count per buffer when clamping (c59f200) * **diagnostic:** handle an unknown or missing client (#16242) (1fdbd29) * **diagnostic:** handle diagnostics placed past the end of line (#16095) (a2994c8) * **diagnostic:** make set() go through cache when calling show() (d93f47d) * **diagnostic:** only update decorations for loaded buffers (#15715) (924e8e4) * **diagnostic:** preserve fields from LSP diagnostics via user_data (#15735) (17b7968) * **diagnostic:** remove useless highlight links (#15683) (c13242c), closes #15585 * disable clipboard when test registers (dd63d93) * **docgen:** add tagfunc.lua (0746f00) * **doc:** various fixes #15604 (4eb1ebb) * **eval:** add the vimscript-1 feature to has() (18b32fc) * **eval:** checking for a non-empty string is too strict (#15987) (1dbbaf8) * **eval:** fix has('wsl') #16153 (16d06fa), closes #12642 #16143 * **eval:** fixup for empty modifier in fnamemodify (#16368) (a7ad509), closes #16367 * **extmark:** fix missing virt_lines when using id param of set_extmark (995dbd2) * **extmarks:** splice extmarks on nv_Undo #15920 (e069361) * **fileio:** replace characters over INT_MAX with U+FFFD (#16354) (a2e5c2f), closes #11877 * **float:** fix potential heap corruption in win_redr_border (de670f3) * **float:** redraw if w_border_adj changed (7ff1bc1) * **heath/provider.vim:** using list as string #16007 (5365f24), closes #15988 * **highlight:** remove syncolor.vim, always include syntax colors (9afa0d2), closes #15176 #12573 #15205 * **inccommand:** ignore trailing commands only for *previewed* command #15638 (1f8c91b), closes #8796 #7494 * include ci/ in exported tarball (d6f03aa), closes #15856 * **input:** never reinterpret unmapped ALT- chrods in Terminal mode (#16222) (5ce35ab) * **input:** resolve isolated (non-ALT/META) mappings #13109 (c4857b6), closes #13042 #13086 #15869 * **jobwait:** always drain process event queues #15402 (3c081d0), closes #15349 * **lsp_spec:** tests depended on previous session (069d1de) * **lsp:** accept file URIs without a hostname (a2c2a08) * **lsp:** add done flag to messages returned in util.get_progress_messages() (#15985) (45fa70a) * **lsp:** add placeholder cancel function (#16189) (4da0351) * **lsp:** add textDocument/prepareRename to capability map (#15961) (fcc11d5), closes #15899 * **lsp:** adjust legacy show diagnostic functions to use correct scope (#16106) (dc6c9fe) * **lsp:** allow diagnostic.clear to accept nil bufnr (#15137) (4ed2d4f) * **lsp:** avoid duplicates in client attached buffers (#16099) (c5525f2) * **lsp:** avoid serializing boolean as key (#15810) (96614f8) * **lsp:** change rpc start notify level to warn (#16467) (04c7b55) * **lsp:** change signature of buf_highlight_references (#16345) (eb3d591) * **lsp:** correctly parse LSP snippets #15579 (516775e), closes #15522 * **lsp:** default to UTF-16 in make_position_params (2e3a474) * **lsp:** do not index nil client in progress (#16262) (8f31b21) * **lsp:** do not invoke handlers for unsupported methods (#15926) (d288daa) * **lsp:** don't update active_clients on exit_timeout (#16192) (98f5782) * **lsp:** enable additional capabilities (#15470) (5d63354) * **lsp:** ensure buffers are re-attached on rename (#16266) (ee3a58d) * **lsp:** Ensure users get feedback on references/symbols errors or empty results (256570a) * **lsp:** expose ContentModified error code to callbacks (#15262) (3f09732) * **lsp:** fix cursor row after textEdits (#16038) (bd2f61c) * **lsp:** gracefully handle nil workspaceFolders (#16284) (0ecc58c) * **lsp:** guard textDocument/codeAction command logic #15769 (433bda4) * **lsp:** improve symbols_to_items performance (#16197) (5ad15c9) * **lsp:** Include client name in handler error messages (#15227) (24f2b9e) * **lsp:** pass bufnr for async formatting (#15084) (c31bc6e) * **lsp:** persist diagnostic config for clients (bcc9ba5) * **lsp:** prevent double <text> for cached plaintext markup (910967e) * **lsp:** restore diagnostics extmarks on buffer changes (#15011) (77b33e4) * **lsp:** rewrite incremental sync (#16252) (2ecf0a4) * **lsp:** send buffer contents joined on fileformat-specific linebreak (#16334) (134a638) * **lsp:** send textDocument/didChange for each buffer (#16431) (3451121) * **lsp:** support duplicate params in signature help (#15032) (9132b76) * **lsp:** update lsp-handler signature in call_hierarchy (#15738) (8164adc) * **lsp:** update workspace/applyEdit handler signature (#15573) (3f526fe) * **lua:** fix vim.deepcopy for metatables & cycled tables (#16435) (eb876a0) * **lua:** preserve argument lists which are not lists (6896d22) * **man.vim:** ensure buftype=nofile after :tag or :stag #15675 (29bc648), closes #15650 * **man.vim:** filetype=man is too eager #15488 (2548a9e), closes #15487 #15487 * **mouse:** fix mouse drag positions on multigrid #12667 (0dcfd0e), closes #15091 * **mouse:** correct dragged position in composed layout (810da1a) * **multigrid:** mouse events crash neovim (28ac6c0) * **nvim_open_win:** crash if autocmds delete buffer/window #15549 (0c06da1), closes #15548 * **options:** using :set fillchars should clear local value (7528bce) * prevent K_EVENT from stopping Select mode CTRL-O #15688 (5f144ef) * **provider:** compare versions as number, not string (python 3.10 support) #15937 (e16adbf), closes #14586 * **screen:** make display_tick monotonic up to 2^64. fixes #16152 (9e88c9c) * **screen:** missing search highlights when redrawing from timer #15380 (db695cc), closes #13074 #14064 * **shared:** do not treat empty tables as list in deep extend (#15094) (526fc60) * **sign:** reset auto sign column with minimum in float win minimal style (c8f57f6) * **startup:** init.lua: set $MYVIMRC to absolute path #15748 (c76cddf) * **termdebug:** replace mapset with nvim_set_keymap (#15699) (4d7dcbe) * **termdebug:** replace term_getline with getbufline #15598 (11289ad) * **terminal:** close without ! if the job is stopped (55defa1), closes #4683 * **terminal:** free terminal if close_buffer() closes a closed terminal (#16264) (14def4d) * **test/dumplog:** tostring(rv) before formatting as string (ddaa0cc) * **tests:** use isolated XDG_DATA_HOME in startup tests (8e663e2) * **treesitter:** do not map hl_group when no mapping is set (f489d98) * **treesitter:** run predicates more often in iter_matches (458f2aa) * **tui:** extend smglr ignores to smglp and smgrp (#16239) (3ba800f) * **tui:** remove obsolete $NVIM detection #15791 (4414584), closes #12937 #11390 * **tutor:** formatting, layout #15098 (c52ec8f), closes #15088 * **ui:** use nowait for q mapping in floating window (#16427) (c132144) * **v:lua:** fix emsg when calling v:lua directly (da9005a) * **vim-patch.sh:** run nvim with -u NONE -n #16179 (97ae0ab) * **vim.opt:** vimL map string values not trimmed (#14982) (4906156) * **window:** win_close from other tabpage #15454 (90b2da1), closes #15313 * **windowing:** positioning of relative floats (9065730)
Shougo
pushed a commit
to Shougo/neovim
that referenced
this issue
Dec 1, 2021
Shougo
pushed a commit
to Shougo/neovim
that referenced
this issue
Dec 1, 2021
BREAKING CHANGES * 32-bit windows builds are no longer provided. * **build deps**: use libuv 1.42.0 upstream for WIN32 (neovim#15889) (f6c0a37), closes neovim#15889 * removes Windows 7 support * removes support for mouse and alternate buffers in TUI for Windows 8 and 8.1 * **lsp/diagnostic:** highlight groups and signs for LSP diagnostics renamed (e.g. `LspDiagnosticsDefaultWarning` to `DiagnosticWarn`) (a5bbb93) * **diagnostic:** make DiagnosticChanged a first class autocmd (neovim#16098) (150a592), closes neovim#16098 * `au User LspDiagnosticsChanged` autocommands are not supported. Use the new first-class DiagnosticChanged event instead. * **lua**: `register_keystroke_callback` => `on_key` (neovim#15460) (69fe427) * **note**: this breaking change was included in 0.5.1 FEATURES * **:source, nvim_exec:** defer script item creation until s:var access (da9b0ab) * **:source, nvim_exec:** support script-local variables (d4ed51e), closes neovim#13143 neovim#11507 * **lua:** add trimempty optional parameter to vim.split (5fa26e2) * **lua:** add vim.str_utf_{start,end} (neovim#16129) (2230b57) * **lua:** add vim.str_utf_pos function (d752cbc) * **lsp:** aggregate code actions from all clients (neovim#15121) (c36df20) * **api:** add lua C bindings for xdiff (neovim#14536) (3d3c0c6) * **api:** evaluate statusline string neovim#16020 (9086938) * **api:** named marks set, get, delete neovim#15346 (49fdc62) * **api:** nvim_get_chan_info: include "argv" for jobs neovim#15537 (0603eba), closes neovim#15440 * **api:** win_viewport also sends line_count neovim#15613 (086631c) * **api:** support :terminal input callback in lua (9e41e82) * **ci:** add backport PR action (neovim#14766) (6cc456d) * **decorations:** allow more than one stacked highlight in a virt_text (1495d36) * **decorations:** support virtual lines (392c658) (8d7816c) * **diagnostic:** move vim.lsp.diagnostic to vim.diagostic and support other sources (a5bbb93) * **diagnostic:** add 'prefix' option to open_float (neovim#16321) (3c74ba4) * **diagnostic:** add option to include diagnostic source (d43151e) * **diagnostic:** allow 'prefix' option to return highlight (cc48837) * **diagnostic:** allow customized diagnostic messages (neovim#15742) (d999c96) * **diagnostic:** match(), tolist(), fromlist() neovim#15704 (e61ea77) * **diagnostic:** update jumplist on goto_next/prev (neovim#15942) (b55944e) * **diagnostic:** use `scope = 'line'` by default for `open_float()` (neovim#16456) (217f9f8), closes neovim#16453 * **diagnostic:** support severity_sort (32c0631) * **checkhealth:** provide function for command line completion (8b43b07) * **f_chansend:** support Blob data argument (7e9ea08) * **job:** add parameter to close stdin (eb7f24b) * **keywordprg:** use :terminal for external commands neovim#15398 (a90513c), closes neovim#2995 neovim#2761 * **lsp:** add 'focus' option to open_floating_preview (neovim#16465) (fff8827) * **lsp:** add a registry for client side code action commands (6c03601) * **lsp:** add client command support to codelens (neovim#15820) (19a77cd) * **lsp:** add codeAction/resolve support (neovim#15818) (ec4731d) * **lsp:** add exit_timeout flag (neovim#16070) (80456cf) * **lsp:** add formatexpr (neovim#16186) (52fa1d2) * **lsp:** add lsp healthcheck (e268026) * **lsp:** add per-client commands (neovim#16101) (519d8de) * **lsp:** add tagfunc (neovim#16103) (f940e7a) * **lsp:** add warning message for large log size (e6777a7) * **lsp:** allow configuring zindex for floating windows (neovim#15086) (c487a73) * **lsp:** allow diagnostics to be disabled for a buffer (neovim#15134) (1aeb945) * **lsp:** allow root_dir to be nil (neovim#15430) (ff0833c) * **lsp:** highlight active parameter in signature help (neovim#15018) (af26371) * **lsp:** improve vim.lsp.util.apply_text_edits (neovim#15561) (41cfba6) * **lsp:** include original request params in handler ctx (187579f) * **lsp:** jump to diagnostics by position (neovim#14795) (ea39ff5) * **lsp:** Make line diagnostics display prettier (e43dbfd) * **lsp:** make list handlers configurable (neovim#15199) (3e00d4f) * **lsp:** support textDocument/prepareRename (neovim#15514) (c1f573f) * **lsp:** use uv_spawn to check if server executable (neovim#16430) (1a60580) * **lsp:** use vim.ui.select() in codelenses (neovim#16004) (e7ea54a) * **lsp:** utilize textEdit.range for startbyte in omnifunc (neovim#15957) (e9d6f7c) * **lua:** add lua-cjson as vendored dependency (8decc9f) * **lua:** add vim.mpack for msgpack support in lua (eaf661d) * **lua:** allow passing handles to vim.b/w/t (6c5e7bd) * **lua:** convert binary string with NULs to Blob (de9df82) * **lua:** document support of packages with v:lua syntax (9dd371b) * **lua:** enable stack traces in error output (neovim#16228) (03b805a) * **lua:** expose lua-cjson as vim.json (30fed27) * **lua:** make vim.mpack support vim.NIL and vim.empty_dict() (0f59666) * **match:** allow hl group to be defined after :match command (fca52f5) * **msgpack:** convert Blobs to BIN strings (af6f454) * **health:** support lua healthchecks (9249dcd), closes neovim#15632 * **shada:** restore Blob globals properly (ef729fb) * **terminal:** TermClose: set exit code in v:event.status neovim#15406 (50b30de), closes neovim#4713 * **treesitter:** add next, prev sibling method (1400841) * **treesitter:** allow to set highlight priority for queries (242608e) * **ui:** add `opt.kind` to `vim.ui.select` (neovim#15838) (7ae86c1) * **ui:** add vim.ui.input and use in lsp rename (neovim#15959) (16d4af6) * **ui:** add vim.ui.select and use in code actions (neovim#15771) (63fde08) * **vim script:** support calling v:lua as a method (b2994e3) CHANGES * **defaults:** auto-create backup dir (4600193) * **defaults:** inccommand=nosplit neovim#15395 (7215d35) * **defaults:** set undo points in <C-U> and <C-W> (neovim#15400) (2cb8db3) * **defaults:** limit syntax cost on CmdwinEnter neovim#15401 (622a36b), closes neovim#6289 neovim#6399 * **defaults:** map CTRL-L to search highlights, update diffs neovim#15385 (0aa8128) * **defaults:** map Y to y$ neovim#13268 (5a111c1), closes neovim#416 neovim#6289 * **defaults:** remove 'options' from viewoptions neovim#15397 (3954537), closes neovim#6289 * **defaults:** set hidden (f6c72b7) * **defaults:** set nojoinspaces (d417e67) * **defaults:** switchbuf=uselast neovim#15394 (4ba7495) * **runtime:** add packages as `"/pack/*/start/*"` patterns to &rtp (9df7e02) * **startup:** load builtin plugins with --clean neovim#15893 (c7a63f3), closes neovim#15605 * **terminal:** set cursorlineopt=number in terminal mode (neovim#15493) (c61a386) * **window:** skip non-focusable floats for :windo (neovim#15378) (e8631cb) PERFORMANCE IMPROVEMENTS * **api:** avoid spurious allocations when converting small objects (705e8f1) * **highlight:** use a hashtable for highlight group names (bb4b4d7) * **lua:** optimize vim.deep_equal neovim#15236 (4b452d4) * **lua:** don't use regexes inside lua require'mod' (ea2023f), closes neovim#15147 neovim#15497 * **lsp:** improve json deserialization performance (neovim#15854) (912a6e5) * **map:** reduce double pointer indirection to single pointer indirection (9e651a9) * **treesitter:** avoid string lookup of highlight name in hot loop (2460f0a) FIXES * **:source, nvim_exec:** handle Vimscript line continuations neovim#14809 (6188926), closes neovim#14807 * **:source:** copy curbuf lines to memory before sourcing neovim#15111 (afdc9e6) * allow str_utfindex second argument to be an explicit nil (neovim#16448) (512ec46) * **api:** fix crash after set_option_value_for() neovim#15390 (8b0e6cc), closes neovim#14097 neovim#13577 * **api:** fix nvim_buf_set_extmark (2338345) * **autocmd:** fix conditions in block_autocmds, unblock_autocmds neovim#15372 (29712ae), closes neovim#6279 * **buffer_updates:** cleanup test behavior (54b2c68) * **buffer_updates:** handle :delete of the very last line in buffer (8335e26) * **buffer_updates:** handle :sort of already sorted buffer (ef687d3) * **buffer_updates:** make `lockmarks` not affect extmarks and buffer updates. fixes neovim#12861 (7d171b1) * **bufupdates:** send correct updates for visual paste (1423146) * **build:** add an env var to re-enable the colors (5087347) * **build:** call find_package(Threads) before using its variables (f446ab3) * **build:** export symbols on Windows (aa644b7) * **build:** fix build failure in MinGW (0503e17) * **build:** make vendored libmpack and libmpack-lua build properly (2a08aef) * **channel:** throw error if sending to internal channel w/o terminal (3b89fee) * **checkhealth:** duplicate checks if module name has "-" neovim#15935 (a36c6e5) * **checkhealth:** mitigate issues with duplicate healthchecks neovim#15919 (acd5e83), closes neovim#15259 * **ci:** disable broken test on openbsd on all CI due to resource constraints (a3e2636) * **ci:** re-run GHA for ready_for_review events (neovim#15377) (c6ef956) * **decorations:** crash when :bdelete (extmark_free_all) after clear_namespace (cd353aa), closes neovim#15212 * **defaults:** "syntax sync maxlines=1" on CmdwinEnter neovim#15552 (5f8518b), closes neovim#15401 * **defaults:** do not map Y in visual-mode neovim#15387 (54726e8), closes neovim#13268 * **diagnostic:** allow floats to be focusable (neovim#16093) (427bac6) * **diagnostic:** change default severity_sort order (938ed45) * **diagnostic:** clamp line numbers in setqflist and setloclist (5b0d8f8) * **diagnostic:** correctly handle folder level diagnostics (f87779a) * **diagnostic:** deepcopy diagnostics before clamping line numbers (2abc799) * **diagnostic:** do not focus floats in goto functions (neovim#16433) (b5b025f) * **diagnostic:** don't clamp line numbers in setqflist (0341c68) * **diagnostic:** don't return nil when callers expect a table (neovim#15765) (057606e) * **diagnostic:** don't use nil col if missing from qflist (neovim#16357) (5e46f64) * **diagnostic:** error on invalid severity value (neovim#15965) (d5dd0aa) * **diagnostic:** fix navigation with diagnostics placed past end of line (34bb5fa) * **diagnostic:** fix option resolution in open_float (neovim#16229) (fd34784) * **diagnostic:** fix wrong data type in setqflist() (3fd1450) * **diagnostic:** get line count per buffer when clamping (c59f200) * **diagnostic:** handle an unknown or missing client (neovim#16242) (1fdbd29) * **diagnostic:** handle diagnostics placed past the end of line (neovim#16095) (a2994c8) * **diagnostic:** make set() go through cache when calling show() (d93f47d) * **diagnostic:** only update decorations for loaded buffers (neovim#15715) (924e8e4) * **diagnostic:** preserve fields from LSP diagnostics via user_data (neovim#15735) (17b7968) * **diagnostic:** remove useless highlight links (neovim#15683) (c13242c), closes neovim#15585 * disable clipboard when test registers (dd63d93) * **docgen:** add tagfunc.lua (0746f00) * **doc:** various fixes neovim#15604 (4eb1ebb) * **eval:** add the vimscript-1 feature to has() (18b32fc) * **eval:** checking for a non-empty string is too strict (neovim#15987) (1dbbaf8) * **eval:** fix has('wsl') neovim#16153 (16d06fa), closes neovim#12642 neovim#16143 * **eval:** fixup for empty modifier in fnamemodify (neovim#16368) (a7ad509), closes neovim#16367 * **extmark:** fix missing virt_lines when using id param of set_extmark (995dbd2) * **extmarks:** splice extmarks on nv_Undo neovim#15920 (e069361) * **fileio:** replace characters over INT_MAX with U+FFFD (neovim#16354) (a2e5c2f), closes neovim#11877 * **float:** fix potential heap corruption in win_redr_border (de670f3) * **float:** redraw if w_border_adj changed (7ff1bc1) * **heath/provider.vim:** using list as string neovim#16007 (5365f24), closes neovim#15988 * **highlight:** remove syncolor.vim, always include syntax colors (9afa0d2), closes neovim#15176 neovim#12573 neovim#15205 * **inccommand:** ignore trailing commands only for *previewed* command neovim#15638 (1f8c91b), closes neovim#8796 neovim#7494 * include ci/ in exported tarball (d6f03aa), closes neovim#15856 * **input:** never reinterpret unmapped ALT- chrods in Terminal mode (neovim#16222) (5ce35ab) * **input:** resolve isolated (non-ALT/META) mappings neovim#13109 (c4857b6), closes neovim#13042 neovim#13086 neovim#15869 * **jobwait:** always drain process event queues neovim#15402 (3c081d0), closes neovim#15349 * **lsp_spec:** tests depended on previous session (069d1de) * **lsp:** accept file URIs without a hostname (a2c2a08) * **lsp:** add done flag to messages returned in util.get_progress_messages() (neovim#15985) (45fa70a) * **lsp:** add placeholder cancel function (neovim#16189) (4da0351) * **lsp:** add textDocument/prepareRename to capability map (neovim#15961) (fcc11d5), closes neovim#15899 * **lsp:** adjust legacy show diagnostic functions to use correct scope (neovim#16106) (dc6c9fe) * **lsp:** allow diagnostic.clear to accept nil bufnr (neovim#15137) (4ed2d4f) * **lsp:** avoid duplicates in client attached buffers (neovim#16099) (c5525f2) * **lsp:** avoid serializing boolean as key (neovim#15810) (96614f8) * **lsp:** change rpc start notify level to warn (neovim#16467) (04c7b55) * **lsp:** change signature of buf_highlight_references (neovim#16345) (eb3d591) * **lsp:** correctly parse LSP snippets neovim#15579 (516775e), closes neovim#15522 * **lsp:** default to UTF-16 in make_position_params (2e3a474) * **lsp:** do not index nil client in progress (neovim#16262) (8f31b21) * **lsp:** do not invoke handlers for unsupported methods (neovim#15926) (d288daa) * **lsp:** don't update active_clients on exit_timeout (neovim#16192) (98f5782) * **lsp:** enable additional capabilities (neovim#15470) (5d63354) * **lsp:** ensure buffers are re-attached on rename (neovim#16266) (ee3a58d) * **lsp:** Ensure users get feedback on references/symbols errors or empty results (256570a) * **lsp:** expose ContentModified error code to callbacks (neovim#15262) (3f09732) * **lsp:** fix cursor row after textEdits (neovim#16038) (bd2f61c) * **lsp:** gracefully handle nil workspaceFolders (neovim#16284) (0ecc58c) * **lsp:** guard textDocument/codeAction command logic neovim#15769 (433bda4) * **lsp:** improve symbols_to_items performance (neovim#16197) (5ad15c9) * **lsp:** Include client name in handler error messages (neovim#15227) (24f2b9e) * **lsp:** pass bufnr for async formatting (neovim#15084) (c31bc6e) * **lsp:** persist diagnostic config for clients (bcc9ba5) * **lsp:** prevent double <text> for cached plaintext markup (910967e) * **lsp:** restore diagnostics extmarks on buffer changes (neovim#15011) (77b33e4) * **lsp:** rewrite incremental sync (neovim#16252) (2ecf0a4) * **lsp:** send buffer contents joined on fileformat-specific linebreak (neovim#16334) (134a638) * **lsp:** send textDocument/didChange for each buffer (neovim#16431) (3451121) * **lsp:** support duplicate params in signature help (neovim#15032) (9132b76) * **lsp:** update lsp-handler signature in call_hierarchy (neovim#15738) (8164adc) * **lsp:** update workspace/applyEdit handler signature (neovim#15573) (3f526fe) * **lua:** fix vim.deepcopy for metatables & cycled tables (neovim#16435) (eb876a0) * **lua:** preserve argument lists which are not lists (6896d22) * **man.vim:** ensure buftype=nofile after :tag or :stag neovim#15675 (29bc648), closes neovim#15650 * **man.vim:** filetype=man is too eager neovim#15488 (2548a9e), closes neovim#15487 neovim#15487 * **mouse:** fix mouse drag positions on multigrid neovim#12667 (0dcfd0e), closes neovim#15091 * **mouse:** correct dragged position in composed layout (810da1a) * **multigrid:** mouse events crash neovim (28ac6c0) * **nvim_open_win:** crash if autocmds delete buffer/window neovim#15549 (0c06da1), closes neovim#15548 * **options:** using :set fillchars should clear local value (7528bce) * prevent K_EVENT from stopping Select mode CTRL-O neovim#15688 (5f144ef) * **provider:** compare versions as number, not string (python 3.10 support) neovim#15937 (e16adbf), closes neovim#14586 * **screen:** make display_tick monotonic up to 2^64. fixes neovim#16152 (9e88c9c) * **screen:** missing search highlights when redrawing from timer neovim#15380 (db695cc), closes neovim#13074 neovim#14064 * **shared:** do not treat empty tables as list in deep extend (neovim#15094) (526fc60) * **sign:** reset auto sign column with minimum in float win minimal style (c8f57f6) * **startup:** init.lua: set $MYVIMRC to absolute path neovim#15748 (c76cddf) * **termdebug:** replace mapset with nvim_set_keymap (neovim#15699) (4d7dcbe) * **termdebug:** replace term_getline with getbufline neovim#15598 (11289ad) * **terminal:** close without ! if the job is stopped (55defa1), closes neovim#4683 * **terminal:** free terminal if close_buffer() closes a closed terminal (neovim#16264) (14def4d) * **test/dumplog:** tostring(rv) before formatting as string (ddaa0cc) * **tests:** use isolated XDG_DATA_HOME in startup tests (8e663e2) * **treesitter:** do not map hl_group when no mapping is set (f489d98) * **treesitter:** run predicates more often in iter_matches (458f2aa) * **tui:** extend smglr ignores to smglp and smgrp (neovim#16239) (3ba800f) * **tui:** remove obsolete $NVIM detection neovim#15791 (4414584), closes neovim#12937 neovim#11390 * **tutor:** formatting, layout neovim#15098 (c52ec8f), closes neovim#15088 * **ui:** use nowait for q mapping in floating window (neovim#16427) (c132144) * **v:lua:** fix emsg when calling v:lua directly (da9005a) * **vim-patch.sh:** run nvim with -u NONE -n neovim#16179 (97ae0ab) * **vim.opt:** vimL map string values not trimmed (neovim#14982) (4906156) * **window:** win_close from other tabpage neovim#15454 (90b2da1), closes neovim#15313 * **windowing:** positioning of relative floats (9065730)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
nvim --version: NVIM v0.5.0-23-gafd576ee9vim -u DEFAULTS(version: ) behaves differently? yes$TERM: xterm-256colorSteps to reproduce using
nvim -u NORCDownload:
neovim_enc_bug.txt
Actual behaviour
Expected behaviour
No crash. (Vim 8 does not crash.)
The text was updated successfully, but these errors were encountered: