# Troubleshooting ## Writes are getting stuck **Update: with the new `Value(func(v []byte))` API, this deadlock can no longer happen.** The following is true for users on Badger v1.x. This can happen if a long running iteration with `Prefetch` is set to false, but an `Item::Value` call is made internally in the loop. That causes Badger to acquire read locks over the value log files to avoid value log GC removing the file from underneath. As a side effect, this also blocks a new value log GC file from being created, when the value log file boundary is hit. Please see GitHub issues [#293](https://github.com/dgraph-io/badger/issues/293) and [#315](https://github.com/dgraph-io/badger/issues/315). There are multiple workarounds during iteration: 1. Use `Item::ValueCopy` instead of `Item::Value` when retrieving value. 1. Set `Prefetch` to true. Badger would then copy over the value and release the file lock immediately. 1. When `Prefetch` is false, don't call `Item::Value` and do a pure key-only iteration. This might be useful if you just want to delete a lot of keys. 1. Do the writes in a separate transaction after the reads. ## Writes are really slow Are you creating a new transaction for every single key update, and waiting for it to `Commit` fully before creating a new one? This leads to very low throughput. We've created `WriteBatch` API which provides a way to batch up many updates into a single transaction and `Commit` that transaction using callbacks to avoid blocking. This amortizes the cost of a transaction really well, and provides the most efficient way to do bulk writes. ```go wb := db.NewWriteBatch() defer wb.Cancel() for i := 0; i < N; i++ { err := wb.Set(key(i), value(i), 0) // Will create txns as needed. handle(err) } handle(wb.Flush()) // Wait for all txns to finish. ``` Note that `WriteBatch` API doesn't allow any reads. For read-modify-write workloads, you should be using the `Transaction` API. ## I don't see any disk writes If you're using Badger with `SyncWrites=false`, then your writes might not be written to value log and won't get synced to disk immediately. Writes to LSM tree are done in-memory first, before they get compacted to disk. The compaction would only happen once `BaseTableSize` has been reached. So, if you're doing a few writes and then checking, you might not see anything on disk. Once you `Close` the database, you'll see these writes on disk. ## Reverse iteration doesn't produce the right results Just like forward iteration goes to the first key which is equal or greater than the SEEK key, reverse iteration goes to the first key which is equal or lesser than the SEEK key. Therefore, SEEK key would not be part of the results. You can typically add a `0xff` byte as a suffix to the SEEK key to include it in the results. See the following issues: [#436](https://github.com/dgraph-io/badger/issues/436) and [#347](https://github.com/dgraph-io/badger/issues/347). ## Which instances should I use for Badger? We recommend using instances which provide local SSD storage, without any limit on the maximum IOPS. In AWS, these are storage optimized instances like i3. They provide local SSDs which clock 100K IOPS over 4KB blocks easily. ## I'm getting a closed channel error ```sh panic: close of closed channel panic: send on closed channel ``` If you're seeing panics like this, it is because you're operating on a closed DB. This can happen, if you call `Close()` before sending a write, or multiple times. You should ensure that you only call `Close()` once, and all your read/write operations finish before closing. ## Are there any Go specific settings that I should use? We _highly_ recommend setting a high number for `GOMAXPROCS`, which allows Go to observe the full IOPS throughput provided by modern SSDs. In Dgraph, we have set it to 128. For more details, [see this thread](https://groups.google.com/d/topic/golang-nuts/jPb_h3TvlKE/discussion). ## Are there any Linux specific settings that I should use? We recommend setting `max file descriptors` to a high number depending upon the expected size of your data. On Linux and Mac, you can check the file descriptor limit with `ulimit -n -H` for the hard limit and `ulimit -n -S` for the soft limit. A soft limit of `65535` is a good lower bound. You can adjust the limit as needed. ## I see "manifest has unsupported version: X (we support Y)" error This error means you have a badger directory which was created by an older version of badger and you're trying to open in a newer version of badger. The underlying data format can change across badger versions and users have to migrate their data directory. Badger data can be migrated from version X of badger to version Y of badger by following the steps listed below. Assume you were on badger v1.6.0 and you wish to migrate to v2.0.0 version. 1. Install Badger version v1.6.0 - `cd $GOPATH/src/github.com/dgraph-io/badger` - `git checkout v1.6.0` - `cd badger && go install` This should install the old Badger binary in your `$GOBIN`. 2. Create Backup - `badger backup --dir path/to/badger/directory -f badger.backup` 3. Install Badger version v2.0.0 - `cd $GOPATH/src/github.com/dgraph-io/badger` - `git checkout v2.0.0` - `cd badger && go install` This should install the new Badger binary in your `$GOBIN`. 4. Restore data from backup - `badger restore --dir path/to/new/badger/directory -f badger.backup` This creates a new directory on `path/to/new/badger/directory` and adds data in the new format to it. NOTE - The preceding steps shouldn't cause any data loss but please ensure the new data is valid before deleting the old Badger directory. ## Why do I need gcc to build badger? Does badger need Cgo? Badger doesn't directly use Cgo but it relies on https://github.com/DataDog/zstd library for zstd compression and the library requires [`gcc/cgo`](https://pkg.go.dev/cmd/cgo). You can build Badger without Cgo by running `CGO_ENABLED=0 go build`. This builds Badger without the support for ZSTD compression algorithm. As of Badger versions [v2.2007.4](https://github.com/dgraph-io/badger/releases/tag/v2.2007.4) and [v3.2103.1](https://github.com/dgraph-io/badger/releases/tag/v3.2103.1) the DataDog ZSTD library was replaced by pure Golang version and Cgo is no longer required. The new library is [backwards compatible in nearly all cases](https://discuss.hypermode.com/t/use-pure-go-zstd-implementation/8670/10): Yes they're compatible both ways. The only exception is 0 bytes of input which gives 0 bytes output with the Go zstd. But you already have the zstd.WithZeroFrames(true) which wraps 0 bytes in a header so it can be fed to DD zstd. This is only relevant when downgrading.