-
Notifications
You must be signed in to change notification settings - Fork 186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documentation / Examples? Multiple Keyspaces (dynamic) using same session/cluster connection #462
Comments
Hi @thunderstumpges, The keyspace string you provide as an argument to the builder is used to create the original keyspace if it doesn't exist. Assuming this is not something you require, all you need to do is to provide a different The builder no longer caches multiple sessions as it doesn't really need to anymore since phantom prefixes every single query with the necessary keyspace. I will think about a clean way to achieve this and possibly providing an internal solution, as you point out it's not a standard use case so we don't normally consider it. Regards. |
So is my first referenced documentation above not accurate anymore? As I see it, currently there is an entirely new session + cluster + connection made for every KeySpaceDef instance. This is exemplified by my code which ends up creating new sessions and cluster connections over and over and over.... If the If not |
Well the more I look into the phantom code, the more I am realizing that I need my own For now I guess I'm looking at my own implementations of each of these classes, plus some logic that supplies a single thanks again, |
Ugh. Well I've now spent too many hours trying to get this working... ran into another wall where BTW, while going through all of this, I noticed the following quote in the java-driver documentation:
However it appears that even in the best / primary use case scenario (static object/val members for each connector / OK, I'm done for tonight I promise! P.S> if you are interested in my late-night stab at a /**
* This implementation caches `Cluster` instances that have the same name. A clusterName in ClusterBuilder is required, or a
* new one will be created every time (probably not what you want)
*/
class SmartSessionProvider(val space: KeySpace, builder: ClusterBuilder) extends SessionProvider {
val session = SmartSessionProvider.session(space.name, builder)
val cluster: Cluster = session.getCluster
}
object SmartSessionProvider {
private val clusterSessions = new mutable.HashMap[String, Session] with mutable.SynchronizedMap[String, Session]
private val keyspaceSessions = new mutable.HashMap[String, Session] with mutable.SynchronizedMap[String, Session]
private val CLUSTER_ID = new AtomicInteger(0)
private def session(keySpace: String, builder: ClusterBuilder) : Session = {
val jBuilder = builder(Cluster.builder)
.withoutJMXReporting().withoutMetrics() // TODO: do we want this?
// make sure we have a name
if (jBuilder.getClusterName == null) jBuilder.withClusterName("cluster" + CLUSTER_ID.incrementAndGet)
// get the existing global session for this cluster or else construct a new one
val session = clusterSessions.getOrElseUpdate(jBuilder.getClusterName, {
jBuilder.build().connect
})
// only auto-create the keyspace once, return the single session for the cluster
keyspaceSessions.getOrElseUpdate(keySpace, {
initKeySpace(session, keySpace)
})
}
/**
* Initializes the keySpace with the given name on
* the specified Session.
*/
private[this] def initKeySpace(session: Session, keySpace: String): Session = blocking {
blocking {
session.execute(s"CREATE KEYSPACE IF NOT EXISTS $keySpace WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor' : 3};")
}
session
}
} |
Hi @thunderstumpges, We are working on an internal update to this, there may be a nicer way to add this at framework level. The problem you will be dealing with is mostly the implicit scope that will be provided by the keyspace, and that's not super easy to handle given it's used literally everywhere. It should still be possible to fix this and give better support, even if this is as you say a fairly niche situation. Regards. |
Thanks @alexflav23. While you mull this over internally, please also consider formalizing the SessionProvider trait so that someone could provide their own implementation. Also, it seems like at some point along the way Connector and KeySpaceDef got muddled together. I see several places in the code that have a variable or parameter called 'connector' and they take a concrete type of KeySpaceDef. Might clean up the abstraction layer if these concepts were formalized a little more. And finally, if none of my quick fixes seem to strike a chord with you guys, how long do you think this internal update might take? We are in a project that needs to continue to progress. I need to weigh my options and this is going to be blocking me in short time. Thanks! |
HI @thunderstumpges, In all honesty, the multiple keyspace thing looks very very dangerous from a distance. I don't know the specifics of what you are doing internally, but this is the first I hear of such an approach undertaken. I can understand you have different keyspaces where you store the same data, but I don't really know why you couldn't simply have a composite key doing the exact same thing and achieving a better distribution of your data over token ranges. Assuming uniform distribution, you'd probably fair a lot better. I doubt Cassandra optimizes heavily for a lot of keyspaces and at the same time if the settings with which they are created are identical, it also makes very little sense to have the keyspaces co-existing altogether. So my gut feeling is that there is a better and simpler way to do this. Regards. |
@alexflav23 We have the same use case too. Has there been any updates on this. Some examples of why we want to create several key keyspaces
|
I'm not certain this is a bug in the code, but I feel like I've been mislead with the documentation and the behavior doesn't match what I thought I'd expect. I am using v1.22.0
First off, my use-case I think is a bit non-standard so that could be part of the problem. I have a consistent schema (Tables/Databases) that exist in multiple keyspaces (of which I don't know ahead of time, and aren't static, but are finite and in the tens max).
I have tried to manipulate examples I've seen which tend to create a static (object) connector for a single keyspace, and instead provide a connector function that will dynamically return a keyspace/session/connector to the same cluster based on a keyspace name.
So, some examples of wordings that make me think this should be easy/possible:
From A series on phantom: Part 1 Getting Started With Phantom:
From KeyspaceBuilder.scala:
and from DefaultSessionProvider.scala
Now, what I have tried, based on examples, and the comments and documentation above, is I have a single instance of KeyspaceBuilder, and I call the .keySpace(keyspace_name) function on it whenever I need a connector for that keyspace (rather than a static val connector):
Now, I can see from tests that this doesn't work, and from what I can tell, it's creating a new cluster, session, etc for every instance of KeySpaceBuilder.keySpace() => KeySpaceDef => DefaultSessionProvider => Session
Am I really expected to create my own Dictionary/Map of keyspace names to instances of KeySpaceBuilder/KeySpaceDef and return the correct one based on the keyspace name? If so, it would be nice if the documentation was clearer on this point. I thought for sure that was handled by phantom.
Thanks in advance!
The text was updated successfully, but these errors were encountered: