Troubleshooting

Something went wrong with my GameServer

If there is something going wrong with your GameServer, there are a few approaches to determining the cause:

Run with the local SDK server

A good first step for seeing what may be going wrong is replicating the issue locally. To do this you can take advantage of the Agones local SDK server , with the following troubleshooting steps:

1. Run your game server as a local binary against the local SDK server
2. Run your game server container against the local SDK server. It’s worth noting that running with docker run --network=host ... can be an easy way to allow your game server container(s) access to the local SDK server)

At each stage, keep an eye on the logs of your game server binary, and the local SDK server, and ensure there are no system errors.

Run as a GameServer rather than a Fleet

A Fleet will automatically replace any unhealthy GameServer under its control - which can make it hard to catch all the details to determine the cause.

To work around this, instantiate a single instance of your game server as a
GameServer within your Agones cluster.

This GameServer will not be replaced if it moves to an Unhealthy state, giving you time to introspect what is going wrong.

Introspect with Kubernetes tooling

There are many Kubernetes tools that will help with determining where things have potentially gone wrong for your game server. Here are a few you may want to try.

kubectl describe

Depending on what is happening, you may want to run kubectl describe <gameserver name> to view the events that are associated with that particular GameServer resource. This can give you insight into the lifecycle of the GameServer and if anything has gone wrong.

For example, here we can see where the simple-game-server example has been moved to the Unhealthy state due to a crash in the backing GameServer Pod container’s binary.

kubectl describe gs simple-game-server-zqppv

The backing Pod has the same name as the GameServer - so it’s also worth looking at the details and events for the Pod to see if there are any issues there, such as restarts due to binary crashes etc.

For example, you can see the restart count on the us-docker.pkg.dev/agones-images/examples/simple-game-server:0.38 container is set to 1, due to the game server binary crash

kubectl describe pod simple-game-server-zqppv

Finally, you can also get the logs of your GameServer Pod as well via kubectl logs <pod name> -c <game server container name>, for example:

kubectl logs simple-game-server-zqppv -c simple-game-server

The above commands will only give the most recent container’s logs (so we won’t get the previous crash), but you can use kubectl logs --previous=true simple-game-server-zqppv -c simple-game-server to get the previous instance of the containers logs, or use your Kubernetes platform of choice’s logging aggregation tools to view the crash details.

kubectl events

The “Events” section that is seen at the bottom of a kubectl describe is backed an actual Event record in Kubernetes, which can be queried - and is general persistent for an hour after it is created.

Therefore, even a GameServer or Pod resource is no longer available in the system, its Events may well be.

kubectl get events can be used to see all these events. This can also be grepped with the GameServer name to see all events across both the GameServer and its backing Pod, like so:

kubectl get events | grep simple-game-server-v992s-jwpx2

Other techniques

For more tips and tricks, the Kubernetes Cheatsheet: Interactive with Pods also provides more troubleshooting techniques.

How do I see the logs for Agones?

If something is going wrong, and you want to see the logs for Agones, there are potentially two places you will want to check:

1. The controller: assuming you installed Agones in the agones-system namespace, you will find that there is a single pod called agones-controller-<hash> (where hash is the unique code that Kubernetes generates) that exists there, that you can get the logs from. This is the main controller for Agones, and should be the first place to check when things go wrong.
   1. To get the logs from this controller run:
      kubectl logs --namespace=agones-system agones-controller-<hash>
2. The SDK server sidecar: Agones runs a small gRPC + http server for the SDK in a container in the same network namespace as the game server container to connect to via the SDK.
   The logs from this SDK server are also useful for tracking down issues, especially if you are having trouble with a particular GameServer.
   1. To find the Pod for the GameServer look for the pod with a name that is prefixed with the name of the owning GameServer. For example if you have a GameServer named simple-game-server, it’s pod could potentially be named simple-game-server-dnbwj.
   2. To get the logs from that Pod, we need to specify that we want the logs from the agones-gameserver-sidecar container. To do that, run the following:
      kubectl logs simple-game-server-dnbwj -c agones-gameserver-sidecar

Agones uses JSON structured logging, therefore errors will be visible through the "severity":"info" key and value.

Enable Debug Level Logging for the SDK Server

By default, the SDK Server binary is set to an Info level of logging.

You can use the sdkServer.logLevel to increase this to Debug levels, and see extra information about what is happening with the SDK Server that runs alonside your game server container(s).

See the GameServer reference for configuration details.

Enable Debug Level Logging for the Agones Controller

By default, the log level for the Agones controller is “info”. To get a more verbose log output, switch this to “debug” via the agones.controller.logLevel Helm Configuration parameters at installation.

The Feature Flag I enabled/disabled isn’t working as expected

It’s entirely possible that Alpha features may still have bugs in them (They are alpha after all 😃), but the first thing to check is what the actual Feature Flags states were passed to Agones are, and that they were set correctly.

The easiest way is to check the top info level log lines from the Agones controller.

For example:

$ kubectl logs -n agones-system agones-controller-7575dc59-7p2rg  | head
{"filename":"/home/agones/logs/agones-controller-20220615_211540.log","message":"logging to file","numbackups":99,"severity":"info","source":"main","time":"2022-06-15T21:15:40.309349789Z"}
{"logLevel":"info","message":"Setting LogLevel configuration","severity":"info","source":"main","time":"2022-06-15T21:15:40.309403296Z"}
{"ctlConf":{"MinPort":7000,"MaxPort":8000,"SidecarImage":"gcr.io/agones-images/agones-sdk:1.23.0","SidecarCPURequest":"30m","SidecarCPULimit":"0","SidecarMemoryRequest":"0","SidecarMemoryLimit":"0","SdkServiceAccount":"agones-sdk","AlwaysPullSidecar":false,"PrometheusMetrics":true,"Stackdriver":false,"StackdriverLabels":"","KeyFile":"/home/agones/certs/server.key","CertFile":"/home/agones/certs/server.crt","KubeConfig":"","GCPProjectID":"","NumWorkers":100,"APIServerSustainedQPS":400,"APIServerBurstQPS":500,"LogDir":"/home/agones/logs","LogLevel":"info","LogSizeLimitMB":10000},"featureGates":"Example=true\u0026NodeExternalDNS=true\u0026PlayerAllocationFilter=false\u0026PlayerTracking=false","message":"starting gameServer operator...","severity":"info","source":"main","time":"2022-06-15T21:15:40.309528802Z","version":"1.23.0"}
...

The ctlConf section has the full configuration for Agones as it was passed to the controller. Within that log line there is a featureGates key, that has the full Feature Gate configuration as a URL Query String (\u0026 is JSON for &), so you can see if the Feature Gates are set as expected.

I uninstalled Agones before deleted all my GameServers and now they won’t delete

Agones GameServers use Finalizers to manage garbage collection of the GameServers. This means that if the Agones controller doesn’t remove the finalizer for you (i.e. if it has been uninstalled), it can be tricky to remove them all.

Thankfully, if we create a patch to remove the finalizers from all GameServers, we can delete them with impunity.

A quick one liner to do this:

kubectl get gameserver -o name | xargs -n1 -P1 -I{} kubectl patch {} --type=merge -p '{"metadata": {"finalizers": []}}'

Once this is done, you can kubectl delete gs --all and clean everything up (if it’s not gone already).

I’m getting Forbidden errors when trying to install Agones

Ensure that you are running Kubernetes 1.12 or later, which does not require any special clusterrolebindings to install Agones.

If you want to install Agones on an older version of Kubernetes, you need to create a clusterrolebinding to add your identity as a cluster admin, e.g.

# Kubernetes Engine
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin --user `gcloud config get-value account`
# Minikube
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin --serviceaccount=kube-system:default

On GKE, gcloud config get-value accounts will return a lowercase email address, so if you are using a CamelCase email, you may need to type it in manually.

I’m getting stuck in “Terminating” when I uninstall Agones

If you try to uninstall the agones-system namespace before you have removed all of the components in the namespace you may end up in a Terminating state.

kubectl get ns

Fixing this up requires us to bypass the finalizer in Kubernetes (article link), by manually changing the namespace details:

First get the current state of the namespace:

kubectl get namespace agones-system -o json >tmp.json

Edit the response tmp.json to remove the finalizer data, for example remove the following:

"spec": {
    "finalizers": [
        "kubernetes"
    ]
},

Open a new terminal to proxy traffic:

 kubectl proxy

Now make an API call to send the altered namespace data:

 curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/agones-system/finalize

You may need to clean up any other Agones related resources you have in your cluster at this point.