post: How to (not) migrate Graylog to Opensearch (#126)

.dictionary

@@ -46,3 +46,8 @@ OSCP
 JSON
 DockerHub
 starlark
+Graylog
+Opensearch
+Elasticsearch
+EOL
+lifecycle

content/posts/2022/how-to-not-migrate-graylog-to-opensearch/index.md

@@ -0,0 +1,46 @@
+---
+title: "How to (not) migrate Graylog to Opensearch"
+date: 2022-07-14T14:45:51+02:00
+authors:
+  - robert-kaussow
+tags:
+  - sysadmin
+resources:
+  - name: feature
+    src: "images/feature.jpg"
+    params:
+      anchor: Center
+      credits: >
+        [CHUTTERSNAP](https://unsplash.com/@chuttersnap) on
+        [Unsplash](https://unsplash.com/s/photos/fail)
+---
+
+Graylog is a centralized log management solution to capture, store and analyze log files in real-time. Starting with the latest minor release 4.3 Graylog announced to no longer support Elasticsearch (ES) due to licensing and structural changes Elastic introduced in v7.11. For this reason, the last supported ES version is 7.10, which has already reached EOL on May 11, 2022.
+
+<!--more-->
+
+Fortunately, [Graylog](https://docs.graylog.org/) also knows this and recommends users to switch even if it is currently not enforced and ES 7.10 continues to work for now <!-- spellchecker-disable -->[^elastic]<!-- spellchecker-enable -->
+. As you usually don't wants to operate software that no longer receives security updates, I have started to look into a migration and prepared the Container and Ansible setup. My fist mistake on this journey was to believe I can just use the latest Opensearch (OS) release. Had I read the documentation <!-- spellchecker-disable -->[^graylog]<!-- spellchecker-enable -->
+more carefully I would have saved myself a lot of trouble...
+
+Anyway, the actual migration of the cluster from ES v7.10 to OS v2.1 succeeded surprisingly smoothly. Well, almost, after all I had to rewrite the complete Ansible role because OS 2.x has changed almost all configuration parameters and API calls :tada: But as you can imagine, everything explodes while trying to start Graylog again. Dang. Just downgrading Opensearch was also not possible as the cluster and all indices were migrated successfully already. To get it back in a working state I decided to reset the entire cluster and restore the snapshots from the S3 backup repository before I start to start a next try, this time with a supported OS 1.x version :fingers_crossed: At least I have already completed the ES disaster recovery test for this year.
+
+Lessons Learned:
+
+- Read documentations/upgrade instructions more carefully
+- Ensure to have a working backup
+- Test your recovery process frequently to stay calm and comfortable in case of an emergency
+- Test upgrades in a staging environment whenever possible
+
+What annoys me a bit about the whole situation is the back and forth and the rather bad communication in the past from Graylog <!-- spellchecker-disable -->[^gl-github]<!-- spellchecker-enable -->
+. Furthermore, the situation with Opensearch is not really better, as it is unclear if e.g. version 1.3 is still supported or not and a general lifecycle information is still missing <!-- spellchecker-disable -->[^os-github]<!-- spellchecker-enable -->
+.
+
+<!-- spellchecker-disable -->
+
+[^elastic]: https://www.graylog.org/post/graylog-to-add-support-for-opensearch
+[^graylog]: https://docs.graylog.org/docs/migrating-to-opensearch
+[^gl-github]: https://github.com/Graylog2/graylog2-server/issues/11804
+[^os-github]: https://github.com/opensearch-project/project-website/issues/661
+
+<!-- spellchecker-enable -->