I just quickly checked the first example you gave and:

[email protected][frwiki]> select * from content_models;
+----------+------------------------+
| model_id | model_name             |
+----------+------------------------+
|        9 | MassMessageListContent |
|        2 | Scribunto              |
|        6 | css                    |
|        3 | flow-board             |
|        5 | javascript             |
|        7 | json                   |
|       10 | sanitized-css          |
|        8 | text                   |
|        1 | wikitext               |
+----------+------------------------+
9 rows in set (0.002 sec)

What's pending to close?

I am going to decline this for now as this will be taken care of by T384810

Patches are done

In T399927#11041786, @ayounsi wrote:

You can probably skip 2039 for now and jump to 2040 until we figure out what's best for 2039.

In T399927#11039668, @Jhancock.wm wrote:

@Marostegui es2038 is moved, updated, and powered up!

Done

Done

Old 300
API 100

db2218 patched

I am repooling this host as replica and closing this for now.

There are no HW errors on the idrac and the syslog only shows:

There are no HW errors on the idrac and the syslog only shows (before and after the crash):

@Jhancock.wm es2038 is ready for you

db2196 is no longer a master - I will leave it depooled and with notifications disabled for the weekend and will investigate more next week.

Done

I will do an emergency switchover of this host, I don't trust it especially now that we are starting the weekend.

db2196 was rebooted via idrac - it shows no HW on its logs

Thanks, I will get es2038 ready by Monday

In T394487#11031363, @jcrespo wrote:

@Marostegui All backups are now generated with the 10.11 package.

@Jhancock.wm es2037 is ready for you - homer was run.

Done

Done

es2036 done

[   21.582858] bnxt_en 0000:4b:00.0 eno12399np0: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[   21.582868] bnxt_en 0000:4b:00.0 eno12399np0: FEC autoneg off encoding: None

In T399927#11031334, @Jhancock.wm wrote:

@Marostegui today or tomorrow is fine.

Patches are done

In T399927#11028165, @Jhancock.wm wrote:

@Marostegui lemme know when you want to do es2036

Patches done

Patches done

it recovered yes, thank you

Thanks, let's see if traffic goes back to normal

This deployment matches a sudden increase in our mainstash database cluster:
https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=2025-07-21T12:17:48.161Z&to=2025-07-22T10:06:34.613Z&timezone=utc&var-site=eqiad&var-group=core&var-shard=ms1&var-shard=ms2&var-shard=ms3&var-role=$__all

@Jhancock.wm es2035 is ready

@FCeratto-WMF now that the s1 codfw master is switched, remember to apply this schema change there too.

In T400055#11020385, @Lucas_Werkmeister_WMDE wrote:

Well, for now it seems to have gone away again – last seen Jul 21, 2025 @ 13:38:00.096 UTC, which was a few minutes before repooling db1166.

My suspicion at the moment is that dbctl had some kind of rare error condition which caused the DEFAULT key to be missing, but fixed itself on the next run.

Yeah, we'll do one at the time, to be on the safe side. Do you want me to get es2035 ready for tomorrow?

In T400055#11020251, @Lucas_Werkmeister_WMDE wrote:

First seen on Jul 21, 2025 @ 12:17:59.330 UTC. In the SAL, two @Marostegui messages (repooling db1157 and depooling db1166) are suspiciously close to that, and the stack trace says “DB config” – any chance this could be related?