Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LingeringRoundRobinLoadBalancerTest failure #2677

Open
Scottmitch opened this issue Aug 20, 2023 · 5 comments · Fixed by #2732
Open

LingeringRoundRobinLoadBalancerTest failure #2677

Scottmitch opened this issue Aug 20, 2023 · 5 comments · Fixed by #2732
Assignees
@bryce-anderson
Labels
flaky tests Unit tests are flaky

Comments

@Scottmitch
Copy link
Member

local macOS m1 build OpenJDK 17.0.3.7

LingeringRoundRobinLoadBalancerTest > expiringAHostDoesntRaceWithConnectionAdding() FAILED
    java.lang.AssertionError: 
    Expected: an instance of io.servicetalk.client.api.NoAvailableHostException
         but: <io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException: Failed to pick an active host for test-service. Either all are busy, expired, or unhealthy: [Host{lbDescription=RoundRobinLoadBalancer{id=LingeringRoundRobinLoadBalancerTest@5f65640c, targetResource=test-service}, address=address-1, state=EXPIRED, #connections=1}]> is a io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at io.servicetalk.loadbalancer.LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding(LingeringRoundRobinLoadBalancerTest.java:160)
@Scottmitch Scottmitch added the flaky tests Unit tests are flaky label Aug 20, 2023
@Scottmitch
Copy link
Member Author

potentially related #1859

@Scottmitch
Copy link
Member Author

https://github.com/apple/servicetalk/runs/16086253855

LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding()

java.lang.AssertionError: 
Expected: an instance of io.servicetalk.client.api.NoAvailableHostException
     but: <io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException: Failed to pick an active host for test-service. Either all are busy, expired, or unhealthy: [Host{lbDescription=RoundRobinLoadBalancer{id=LingeringRoundRobinLoadBalancerTest@6296e812, targetResource=test-service}, address=address-1, state=EXPIRED, #connections=1}]> is a io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at io.servicetalk.loadbalancer.LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding(LingeringRoundRobinLoadBalancerTest.java:160)

@Scottmitch
Copy link
Member Author

https://github.com/apple/servicetalk/pull/2719/checks?check_run_id=17322302398

LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding()

java.lang.AssertionError: 
Expected: an instance of io.servicetalk.client.api.NoAvailableHostException
     but: <io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException: Failed to pick an active host for test-service. Either all are busy, expired, or unhealthy: [Host{lbDescription=RoundRobinLoadBalancer{id=LingeringRoundRobinLoadBalancerTest@43f5d671, targetResource=test-service}, address=address-1, state=EXPIRED, #connections=1}]> is a io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at io.servicetalk.loadbalancer.LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding(LingeringRoundRobinLoadBalancerTest.java:160)

@Scottmitch
Copy link
Member Author

https://github.com/apple/servicetalk/runs/17378313928

LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding()

java.lang.AssertionError: 
Expected: an instance of io.servicetalk.client.api.NoAvailableHostException
     but: <io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException: Failed to pick an active host for test-service. Either all are busy, expired, or unhealthy: [Host{lbDescription=RoundRobinLoadBalancer{id=LingeringRoundRobinLoadBalancerTest@133a713f, targetResource=test-service}, address=address-1, state=EXPIRED, #connections=1}]> is a io.servicetalk.loadbalancer.RoundRobinLoadBalancer$StacklessNoActiveHostException
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at io.servicetalk.loadbalancer.LingeringRoundRobinLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding(LingeringRoundRobinLoadBalancerTest.java:160)

@bryce-anderson bryce-anderson self-assigned this Oct 4, 2023
@bryce-anderson bryce-anderson mentioned this issue Oct 18, 2023
Merged
bryce-anderson added a commit that referenced this issue Oct 18, 2023
@bryce-anderson
Fix the flaky LingeringRoundRobinLoadBalancerTest (#2732)
61973fe 
Motivation:

The test `expiringAHostDoesntRaceWithConnectionAdding` is flaky but only in CI.
This is because locally we essentially never hit the `else` branch where the
connection was added first but it does happen in CI. The `else` branch of the
test is flawed: the host is marked as `EXPIRED` and thus unhealthy, so we should
always expect a `NoActiveHostException` instead of a `NoAvailableHostException`.

Modifications:

Change the expected exception to NoActiveHostException.

Result:

Less flakiness.
Closes #2677.
@idelpivnitskiy
Copy link
Member

https://github.com/apple/servicetalk/actions/runs/12471261226/job/34807889797?pr=3152

> Task :servicetalk-loadbalancer:test

LingeringNewRoundRobinLoadBalancerTest > expiringAHostDoesntRaceWithConnectionAdding() FAILED
    java.lang.AssertionError: 
    Expected: iterable containing [(hasProperty("key", is "address-1") and hasProperty("value", a collection with size <1>))]
         but: no item was (hasProperty("key", is "address-1") and hasProperty("value", a collection with size <1>))
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at io.servicetalk.loadbalancer.LoadBalancerTestScaffold.assertConnectionCount(LoadBalancerTestScaffold.java:173)
        at io.servicetalk.loadbalancer.LingeringLoadBalancerTest.expiringAHostDoesntRaceWithConnectionAdding(LingeringLoadBalancerTest.java:156)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bryce-anderson bryce-anderson

Labels
flaky tests Unit tests are flaky
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix the flaky LingeringRoundRobinLoadBalancerTest bryce-anderson/servicetalk
3 participants
@bryce-anderson @idelpivnitskiy @Scottmitch