As a kdb+/Q programmer I would say it depends on the type of problem.
For example, when working with arrays of data it certainly is easier to think and write “avg a+b” to add two arrays together and then take the average.
In a non-array programming language you would probably first need to do some bounds checking, then a big for loop, a temporary variable to hold the sum and the count as you loop over the two arrays, etc.
Probably the difference between like 6ish lines of code in some language like C versus the 6 characters above in Q.
But every language has features that help you reason about certain types of problems better. Functional languages with algebraic data types and pattern matching (think OCaml or F#) are nicer than switch statements or big if-else-if statements. Languages with built-in syntactic sugar like async/await are better at dealing with concurrency, etc.
Well no, not in a non-array programming language. In any language that has a semi-decent type/object system and some kind of functional programming support, `avg a+b` would just be `avg(a, b)`, which is not any easier or harder, with an array type defined somewhere. Once you make your basic array operations (Which they have to be made in q anyways, just in the stdlib), you can compose them just like you would in q, and get the same results. All of the bounds checking and for-loops is unnecessary, all you really need are a few HKTs that do fancy maps and reduces, which the most popular languages already have.
A very real example of this is Julia. Julia is not really an array-oriented programming language, it's a general language with a strong type system and decent functional programming facilities, with some syntactic sugar that makes it look like it's a bit array oriented. You could write any Q/k program in Julia with the same complexity and it would not be any more complex. For a decently complex program Julia will be faster, and in every case it will be easier to modify and read and not any harder to write.
I don't know what you mean by the q array operations being defined in the standard library. Yes there are things defined in .q, but they're normally thin wrappers over k which has array operations built in.
I don't consider an interpreted language having operations "built-in" be significantly different from a compiled language having basic array operations in the stdlib or calling a compiled language.
It is syntactically different, not semantically different. If you gave me any reasonable code in k/q I'm pretty confident I could write semantically identical Julia and/or numpy code.
In fact I've seen interop between q and numpy. The two mesh well together. The differences are aesthetic more than anything else.
There are semantic differences too with a lot of the primitives that are hard to replicate exactly in Julia or numpy. That's without mentioning the stuff like tables and IPC, which things like pandas/polars/etc don't really come close to in ergonomics, to me anyway.
Do you have examples of primitives that are hard to replicate? I can't think of many off the top of my head.
> tables and IPC
Sure, kdb doesn't really have an equal, though it is very niche. But for IPC I disagree. The facilities in k/q are neat and simple in terms of setup, but it doesn't have anything better than what you can do with cloudpickle, and the lack of custom types makes effective, larger-scale IPC difficult without resorting to inefficient hacks.
None of the primitives are necessarily too complicated, but off the top of my head things like /: \: (encode, decode), all the forms of @ \ / . etc, don't have directly equivalent numpy functions. Of course you could reimplement the entire language, but that's a bit too much work.
Tables aren't niche, they're very useful! I looked at cloudpickle, and it seems to only do serialisation, I assume you'd need something else to do IPC too? The benefit of k's IPC is it's pretty seamless.
I'm not sure what you mean by inefficient hacks, generally you wouldn't try to construct some complicated ADT in k anyway, and if you need to you can still directly pass a dictionary or list or whatever your underlying representation is.
> None of the primitives are necessarily too complicated, but off the top of my head things like /: \: (encode, decode), all the forms of @ \ / . etc, don't have directly equivalent numpy functions. Of course you could reimplement the entire language, but that's a bit too much work.
@ and . can be done in numpy through ufunc. Once you turn your unary or binary function into a ufunc using food = np.frompyfunc, you then have foo.at(a, np.s_[fancy_idxs], (b?)) which is equivalent to @[a, fancy_idxs, f, b?]. The other ones are, like, 2 or 3 lines of code to implement, and you only ever have to do it once.
vs and sv are just pickling and unpickling.
> Tables aren't niche,
Yes, sorry, I meant that tables are only clearly superior in the q ecosystem in niche situations.
> I looked at cloudpickle, and it seems to only do serialisation, I assume you'd need something else to do IPC too? The benefit of k's IPC is it's pretty seamless.
Python already does IPC nicely through the `multiprocess` and `socket` modules of the standard library. The IPC itself is very nice in most usecases if you use something like multiprocessing.Queue. The thing that's less seamless is that the default pickling operation has some corner cases, which cloudpickle covers.
> Im not sure what you mean by inefficient hacks, generally you wouldn't try to construct some complicated ADT in k anyway, and if you need to you can still directly pass a dictionary or list or whatever your underlying representation is.
It's a lot nicer and more efficient to just pass around typed objects than dictionaries. Being able to have typed objects whose types allow for method resolution and generics makes a lot of code so much simpler in Python. This in turns allows a lot of libraries and tricks to work seamlessly in Python and not in q. A proper type system and colocation of code with data makes it a lot easier to deal with unknown objects - you don't need nested external descriptors to tag your nested dictionary and tell you what it is.
Again, I'm not saying anything is impossible to do, it's just about whether or not it's worth it. 2 or 3 lines for all types for all overloads for all primitives etc adds up quickly.
I don't see how k/q tables are only superior in niche situations, I'd much rather (and do) use them over pandas/polars/external DBs whenever I can. The speed is generally overhyped, but it is significant enough that rewriting something from pandas often ends up being much faster.
The last bits about IPC and typed objects basically boil down to python being a better glue language. That's probably true, but the ethos of array languages tends to be different, and less dependent on libraries.
Which is why C# is the giant ever increasing bag of tricks that it is (unkind people might say bloat…) ;-) Personally, I’m all for this; let me express the problem in whatever way is most natural.
There are limits, of course, and it’s not without downsides. Still, if I have to code in something all day, I’d like that “something” be as expressive as possible.
For some classes of problems that are easily vectorized, using an array-focused language can certainly make thinking about them and their solutions more efficient, since you can abstract over the data structure and iteration details.
As a quant, I used kdb+/q quite a bit for 5+ years for mid-frequency strategies, but as I moved towards higher frequency trading that required calculations on the order book that couldn't be easily or efficiently vectorized, then continuing to use array-focused languages would have only complicated reasoning about those problems.
I went to this tech talk on Dyalog (a modern APL-like language), and the speaker makes the argument that the notation allows certain idioms to be recognized more easily:
This particular snippet was in the context of compilers, but the rest of the talk has more on Dyalog and APL as a system of mathematical notation. The underlying theme is that optimizing mathematical expressions may be easier than optimizing general code.
Hillel Wayne writes about it on his newsletter every once in a while. He's convinced me that he does in fact think through some problems better in array languages but I still can't really conceive of what that experience is like.
if it's something you're interested in trying i'd be happy to point you toward more resources, and i'm sure there are plenty of other arraylang tinkerers reading this thread who could help, too
one nice thing about the array language style is that it's possible to talk about variations on algorithms where the relevant code snippets, being a few characters, fits inline into the discussion; more traditional vertically-oriented languages that take handfuls or dozens of lines to say the same things need to intersperse code display blocks with expository prose
"More efficiently"? Maybe. It opens up a new way to think about solutions to problems. Sometimes those solutions are more efficient, and sometimes they are just different.
It's a useful thing to learn though. And dare I say it, fun. Even if there was zero benefit to it, it'd still be fun. As it turns out, there really are benefits.
For me, the biggest benefit is when I'm working with data interactively. The syntax allows me to do a lot of complex operations on sets of data with only a few characters, which makes you feel like you have a superpower (especially when comparing to someone using Excel to try to do the same thing).
I've found that the challenge is to "think in vector operations" rather than of iterating over the same data. The tricky part is figuring out how to get an operator to do the right thing over an array of stuff on the left hand side and this list/bag/etc of arguments on the right
